A data processing system for video retrieval

Abstract

This application discloses a data processing system for video retrieval, which processes text data information and image data information to be processed to obtain first information; and determines target image data information based on the first information and the image data information to be processed.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data processing system for video retrieval. Background Technology

[0002] With the development of internet technology, the number of videos has exploded. To quickly retrieve videos of interest from this vast resource, video retrieval is necessary. However, existing video retrieval methods primarily rely on keyword matching or video metadata, which leads to inaccurate search results. Summary of the Invention

[0003] This application provides a data processing system for video retrieval.

[0004] In a first aspect, this application provides a method comprising:

[0005] Obtain the text data and image data to be processed;

[0006] The text data and image data to be processed are processed to obtain the first information;

[0007] Based on the first information and the image data to be processed, the target image data information is determined.

[0008] Secondly, this application provides a system comprising:

[0009] The information acquisition module is used to acquire text data and image data to be processed.

[0010] The data processing module is used to process the text data and image data to be processed to obtain the first information;

[0011] The data determination module is used to determine the target image data information based on the first information and the image data information to be processed.

[0012] Thirdly, this application also provides an apparatus, the apparatus comprising:

[0013] One or more processors;

[0014] Memory; and

[0015] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor to implement the methods of any one of the first aspects.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the method in any of the first aspects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of one embodiment of the data processing method provided by the present invention;

[0020] Figure 3 This is a flowchart illustrating a specific embodiment of the present invention for obtaining text data information and image data information to be processed.

[0021] Figure 4 This is a flowchart illustrating a specific embodiment of processing text data information and image data information to be processed, provided in this invention.

[0022] Figure 5 This is a schematic diagram of the target processing model provided in an embodiment of the present invention;

[0023] Figure 6 This is a flowchart illustrating a specific embodiment of determining target image data information provided in this invention.

[0024] Figure 7 This is a schematic block diagram of the data processing system provided in the embodiments of the present invention;

[0025] Figure 8 This is a schematic diagram of an embodiment of the computer device provided in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more features.

[0028] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0029] It should be noted that since the method in this application embodiment is executed in a computer device, the processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It is understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. Specific details will not be elaborated here.

[0030] This application provides a data processing method and system for video retrieval, which will be described in detail below.

[0031] Please see Figure 1 , Figure 1 This is a schematic diagram of a data processing system provided in an embodiment of this application. The data processing system may include a computer device 100, which integrates the data processing system, such as... Figure 1 Computer equipment in the country.

[0032] In this embodiment, the computer device 100 is mainly used to acquire text data information to be processed and image data information to be processed; process the text data information to be processed and image data information to be processed to obtain first information; and determine the target image data information based on the first information and the image data information to be processed, which can improve the accuracy and comprehensiveness of video retrieval results.

[0033] In this embodiment, the computer device 100 can be a standalone server, a server network, or a server cluster. For example, the computer device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0034] It is understood that the computer device 100 used in the embodiments of this application can be a device that includes both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the computer device 100 may be a desktop terminal or a mobile terminal, and may also be one of a mobile phone, tablet computer, laptop computer, etc.

[0035] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of computer devices shown is more or less, for example Figure 1 Only one computer device is shown in the diagram. It is understood that the data processing system may also include one or more other services, which are not limited here.

[0036] In addition, such as Figure 1 As shown, the data processing system may also include a memory 200 for storing data, such as text data information, such as text data information to be processed, first text data information, second text data information, etc., and image data information, such as first image data information, second image data information, etc.

[0037] It should be noted that, Figure 1 The schematic diagram of the data processing system shown is merely an example. The data processing system and scenario described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of data processing systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0038] like Figure 2The diagram shown is a flowchart of an embodiment of the data processing method in this application. The data processing method may include the following steps S201 to S203, as detailed below:

[0039] S201. Obtain the text data information and image data information to be processed.

[0040] In this embodiment, the text data information to be processed is the text data information obtained by fusing the first text data information and the third image data information. The first text data information is the query text input by the user, and the third image data information is the image data information to be retrieved. For example, the text data information to be processed can be represented as: t = {t1, t2, ..., t...} m}, t i This represents the i-th word in the text data to be processed.

[0041] Optionally, the computer device can acquire first text data information and third image data information, and then perform fusion processing on the first text data information and third image data information to obtain text data information to be processed; the computer device can also directly acquire the text data information to be processed from other devices through networks, Bluetooth, etc., and this embodiment does not limit it.

[0042] Furthermore, the image data information to be processed is image data information obtained by segmenting the third image data information. Segmenting the third image data information refers to dividing the continuous frames in the third image data information into segments according to specific rules (e.g., time annotation, scene, etc.), and aggregating at least one first image segment information obtained from the segmentation into an image segment set. The image data information to be processed includes at least one first image segment information, which is a continuous frame extracted from the third image data information. Image data refers to the set of grayscale values ​​of each pixel represented numerically. Image data can be static image data or dynamic video data. For example, the image data information to be processed can be represented as: S = {s1, s2, ..., s...} n}, s j This represents the j-th first image segment information in the image data to be processed.

[0043] Optionally, the computer device can acquire third image data information and then segment the third image data information to obtain image data information to be processed. The computer device can also directly acquire the image data information to be processed from other devices through networks, Bluetooth, etc. This embodiment does not limit this.

[0044] In a specific implementation method, refer to Figure 3As shown, obtaining the text data information and image data information to be processed in step S201 above may include steps S301 to S303, as follows:

[0045] S301, Obtain the first text data information and the third image data information.

[0046] In this embodiment, the first text data information is the query text input by the user obtained by the computer device. The computer device can obtain the first text data information in various ways. For example, the computer device can collect the user's voice information through its own microphone and convert the voice information into text information to obtain the first text data information. The computer device can also receive the first text data information input by the user through input devices such as keyboard, mouse, and touch screen. The computer device can also obtain the first text data information from other devices through networks, Bluetooth, etc. This embodiment does not limit the scope of the first text data information.

[0047] Furthermore, the third image data information is the image data information to be retrieved. Image data refers to the set of gray values ​​of each pixel represented by numerical values. Image data can be static image data or dynamic video data. This embodiment does not limit the scope of the data.

[0048] S302. The third image data information is segmented to obtain the image data information to be processed.

[0049] In one specific embodiment, the image data information to be processed includes at least one first image segment information. The first image segment information is a continuous frame extracted from the image data information to be processed, and each first image segment information covers a specific time interval. For example, the image data information to be processed can be represented as S = {s1, s2, ..., s...} n}, s j This represents the j-th first image segment information in the image data to be processed, where each first image segment information s j It covers a specific time interval.

[0050] S303. The first text data information and the third image data information are fused to obtain the text data information to be processed.

[0051] In this embodiment, the text data information to be processed is the text data information obtained by fusing first text data information and third image data information. In a specific embodiment, the step of fusing the first text data information and third image data information to obtain the text data information to be processed specifically includes: generating corresponding second text data information based on the feature representation of the third image data information, the second text data information being used to characterize the information features of the third image data information; and fusing the second text data information and the first text data information to obtain the text data information to be processed. This application uses the fusion processing of the second text data information and the first text data information to obtain the text data information to be processed, and then performs data processing based on the text data information to be processed. This allows for semantic enhancement of the feature representations of important image segments using the text data information to be processed, thereby improving the accuracy of video retrieval results.

[0052] In another specific embodiment, the step of fusing the first text data information and the third image data information to obtain the text data information to be processed specifically includes: generating corresponding second text data information based on the feature representation of the third image data information, wherein the second text data information is used to characterize the information features of the third image data information; performing information recognition on the first text data information to obtain first candidate text information; performing information recognition on the second text data information to obtain second candidate text information; and fusing the first candidate text information and the second candidate text information to obtain the text data information to be processed.

[0053] In this embodiment, the first candidate text information is key information such as people, scenes, and events identified from the first text data information, and the second candidate text information is key information such as people, scenes, and events identified from the second text data information. This embodiment identifies key information from the first and second text data information, and then performs data processing based on the identified key information, which can filter out irrelevant information in the second and first text data information, further improving the accuracy of video retrieval results.

[0054] Furthermore, fusion processing refers to the process of integrating and merging the first candidate text information and the second candidate text information. In this embodiment, the first candidate text information and the second candidate text information are fused to obtain the text data information to be processed. Data processing is then performed based on the text data information to be processed. This can combine the first candidate text information and the second candidate text information to improve the accuracy of the determined target image data information. Optionally, when fusing the first candidate text information and the second candidate text information, the first candidate text information and the second candidate text information can be directly concatenated, or an intersection operation can be performed on the first candidate text information and the second candidate text information, or a union operation can be performed on the first candidate text information and the second candidate text information. This embodiment does not impose any limitations.

[0055] S202. Process the text data information and image data information to be processed to obtain the first information.

[0056] In this embodiment of the application, the image data information to be processed includes at least one first image segment information. The first information includes a first feature value corresponding to each first image segment information. The first feature value can characterize the degree of correlation between the first image segment information and the text data information to be processed. For example, the first information can be represented as: This represents the first feature value corresponding to the j-th first image segment information.

[0057] In a specific implementation method, refer to Figure 4 As shown, step S202 above processes the text data information and image data information to be processed to obtain the first information, which may include steps S401 to S402, as follows:

[0058] S401. Preprocess the image data information to be processed to obtain preprocessed image data information.

[0059] In this embodiment, after obtaining the text data information and image data information to be processed, the image data information to be processed is preprocessed first, and then the preprocessed image data information and the text data information to be processed are processed. Since the preprocessed image data information has a smaller data volume than the image data information to be processed, processing the preprocessed image data information and the text data information to be processed can reduce the amount of computation during data processing.

[0060] In one specific embodiment, the image data information to be processed includes at least one first image segment information, and the preprocessed image data information includes an image segment representation corresponding to each first image segment information. The step of preprocessing the image data information to be processed to obtain preprocessed image data information specifically includes: for any first image segment information in the image data information to be processed, performing a filtering process on the first image segment information to obtain first image data information corresponding to the first image segment information; performing a segmentation process on the first image data information to obtain second image data information corresponding to the first image segment information; the second image data information includes at least one second image segment information; and performing a filtering process on the second image segment information in the second image data information to obtain preprocessed image data information.

[0061] In this embodiment, when filtering the first image segment information, a fixed number of consecutive image frames can be randomly extracted from the first image segment information, or a fixed number of image frames can be extracted from the first image segment information at fixed intervals. This embodiment does not impose any limitation. For example, the first image segment information s i The corresponding first image data information can be represented as: F i ={f i1 ,f i2 ,…,f ip}, p represents the information of the first image segment s i The number of image frames in the corresponding first image data information.

[0062] Furthermore, segmenting the first image data information refers to dividing the continuous frames in the first image data information into segments according to specific rules (e.g., time annotation, scene, etc.), and aggregating at least one second image segment information obtained from the segmentation into an image segment set. For example, the first image data information F containing p continuous frames... i ={f i1 ,f i2 ,…,f ip The image data F can be obtained by uniformly dividing it into k segments. i ′={f i1 ′,f i2 ′,…,f ik ′},f ik ′ represents the information of the k-th second image segment.

[0063] Optionally, when filtering the second image segment information in the second image data information, one frame of image data can be randomly sampled from the second image segment information, or multiple frames of image data can be randomly sampled from the second image segment information, or image data at a fixed position can be sampled from the second image segment information. This embodiment does not impose any limitations. For example, the first image segment information s i The corresponding second image data information can be represented as: F i ′={f i1 ′,f i2 ′,…,f ik ′},f ik ′ represents the information of the k-th second image segment, for F i The information of the first image segment s can be obtained by randomly sampling the information of the second image segment in ''. i The corresponding image segment is represented by: v i ={v i1 ,v i2 ,…,v ik}, v ij Indicates information f from the second image segment ij Image frames obtained by sampling from ''.

[0064] S402. Based on the text data information to be processed and the preprocessed image data information, first information is obtained. The first information is used to characterize the degree of correlation between the image data information to be processed and the text data information to be processed.

[0065] In one specific embodiment, the first information is obtained by processing the text data information to be processed and the preprocessed image data information through a target processing model, referring to... Figure 5 As shown, the target processing model includes a first feature extraction module, a second feature extraction module, and an information determination module. The input of the first feature extraction module is configured to receive preprocessed image data, the input of the second feature extraction module is configured to receive text data to be processed, and the outputs of the first and second feature extraction modules are respectively connected to the input of the information determination module.

[0066] Furthermore, the first feature extraction module is configured to extract features from the preprocessed image data to obtain first feature information; the second feature extraction module is configured to extract features from the text data to be processed to obtain second feature information; and the information determination module is configured to determine first information based on the first and second feature information. This embodiment utilizes the first feature extraction module to extract representative image features and the second feature extraction module to accurately extract semantic features, enabling efficient matching of text and image features and accurate identification of associated image segments.

[0067] In one specific embodiment, reference continues to be made to... Figure 5 As shown, the first feature extraction module includes: a first feature extraction unit, a normalization unit, a feature mapping unit, and a pooling unit. The input of the first feature extraction unit is configured to receive preprocessed image data information. The output of the first feature extraction unit is connected to the input of the normalization unit, the output of the normalization unit is connected to the input of the feature mapping unit, and the output of the feature mapping unit is connected to the input of the pooling unit. The first feature extraction unit is configured to extract features from the preprocessed image data information to obtain third feature information; the normalization unit is configured to normalize the third feature information to obtain fourth feature information; the feature mapping unit is configured to perform feature mapping on the fourth feature information to obtain fifth feature information; and the pooling unit is configured to pool the fifth feature information to obtain the first feature information.

[0068] Optionally, continue to refer to Figure 5 As shown, the first feature extraction unit includes a first convolutional unit, a feature processing unit, and a first encoding unit. The input of the first convolutional unit is configured to receive preprocessed image data information, and the output of the first convolutional unit is connected to the input of the feature processing unit. The output of the feature processing unit is connected to the input of the first encoding unit. The first convolutional unit is configured to perform a convolution operation on the preprocessed image data information to obtain tenth feature information; the feature processing unit is configured to perform a flattening operation on the tenth feature information to obtain eleventh feature information; and the first encoding unit is configured to extract features from the eleventh feature information to obtain third feature information.

[0069] In this embodiment, the feature processing unit can convert multidimensional input data into a one-dimensional vector, thereby facilitating feature extraction by the first encoding unit. For example, the first convolutional unit converts preprocessed image data information into a corresponding frame sequence, the feature processing unit cuts each frame sequence into 7*7 data blocks, and then flattens the cut 7*7 data blocks into 49*1 region features, which are then input into the first encoding unit for feature extraction.

[0070] Furthermore, the first convolutional unit can be a 3D convolutional layer. For example, the first convolutional unit can be a 3D convolutional layer with a kernel size of 3×3×3, a 7×7×7, or an 11×11×11. This embodiment does not impose any limitations. The first encoding unit can be constructed based on a Transformer encoder. The number of the first encoding units can be set according to actual needs. For example, the number of the first encoding units in the first feature extraction unit can be 8, 12, 16, etc. This embodiment does not impose any limitations.

[0071] Optionally, the feature mapping unit includes: a first linear mapping unit, an activation unit, and a second linear mapping unit. The input of the first linear mapping unit is configured to receive fourth feature information, and the output of the first linear mapping unit is connected to the input of the activation unit, which in turn is connected to the input of the second linear mapping unit. The first linear mapping unit is configured to perform linear mapping processing on the fourth feature information to obtain twelfth feature information; the activation unit is configured to perform nonlinear mapping processing on the twelfth feature information to obtain thirteenth feature information; and the second linear mapping unit is configured to perform linear mapping processing on the thirteenth feature information to obtain fifth feature information. In a specific embodiment, the linear mapping processing can employ a three-dimensional mapping method. The three-dimensional linear mapping uses a kernel [t×h×w] as a linear three-dimensional convolution, which can capture the spatiotemporal information of the preprocessed image data, where t, h, and w represent time, height, and width, respectively.

[0072] Optionally, the pooling unit can aggregate the features of the image frame to obtain the image feature representation corresponding to each first image segment. The pooling unit can be constructed based on an average pooling layer or a global pooling layer; this embodiment does not impose any limitations. The activation unit can be constructed based on the GELU activation function, the sigmoid activation function, or the ReLU activation function; this embodiment does not impose any limitations.

[0073] In one specific embodiment, reference continues to be made to... Figure 5 As shown, the second feature extraction module includes a second feature extraction unit and a second encoding unit. The input of the second feature extraction unit is configured to receive text data to be processed, and the output of the second feature extraction unit is connected to the input of the second encoding unit. The second feature extraction unit is configured to perform word embedding processing on the text data to be processed to obtain the fourteenth feature information; the second encoding unit is configured to extract features from the fourteenth feature information to obtain the second feature information. The second encoding unit can be constructed based on a Transformer encoder, and the number of second encoding units can be set according to actual needs. For example, the number of second encoding units in the second feature extraction module can be 8, 12, 16, etc., but this embodiment does not impose a limitation.

[0074] In one specific embodiment, the image data information to be processed includes at least one first image segment information, and the first feature information includes an image feature representation corresponding to each first image segment information. The step of the information determination module determining the first information based on the first feature information and the second feature information specifically includes: calculating and processing the image feature representation and the second feature information corresponding to each first image segment information to obtain the first similarity information corresponding to each first image segment information; and performing information mapping processing on the first similarity information to obtain the first information.

[0075] In one specific embodiment, the step of calculating and processing the image feature representation and second feature information corresponding to each first image segment information to obtain the first similarity information corresponding to each first image segment information specifically includes: transposing the second feature information to obtain the sixth feature information; calculating and processing the image feature representation and the sixth feature information corresponding to each first image segment information to obtain the seventh feature information corresponding to each first image segment information; calculating and processing the image feature representation corresponding to each first image segment information to obtain the eighth feature information corresponding to each first image segment information; calculating and processing the second feature information to obtain the ninth feature information; and calculating and processing the seventh, eighth, and ninth feature information to obtain the first similarity information corresponding to each first image segment information.

[0076] Optionally, calculating the image feature representation corresponding to each first image segment information means calculating the magnitude of the vector of the image feature representation corresponding to each first image segment information, and calculating the second feature information means calculating the magnitude of the vector corresponding to the second feature information. The process of determining the first similarity information can be expressed as follows: s(v i ,t) represents the information of the first image segment v i The first similarity information between the text data t to be processed and the text data t. This represents the sixth feature information. Indicates the information v of the first image segment i The corresponding image feature representation, ||f t ‖ represents the ninth feature information. · represents the eighth feature information, · represents the vector dot product, and ‖·‖ represents the magnitude of the vector.

[0077] Optionally, when performing information mapping processing on the first similarity information as described above, the first similarity information can be mapped to a specific score range. The mapping process for the first similarity information can be expressed as follows: γ varies depending on the specific score range. For example, when the specific score range is 0 to 100, γ = 100, that is...

[0078] In one specific embodiment, the target processing model is trained using a training sample set, which includes sample text data and corresponding sample image data. The training process of the target processing model is as follows: inputting the sample text data and corresponding sample image data into a preset network model; outputting third similarity information between the sample text data and corresponding sample image data from the preset network model; determining a loss value based on the third similarity information and the loss function of the preset network model; if the loss value does not meet the second condition, correcting the model parameters of the preset network model based on a preset parameter learning rate, and continuing to input the sample text data and corresponding sample image data into the preset network model until the loss value meets the second condition. The loss value meeting the second condition can be defined as the loss value being less than a first threshold, or it can be defined as the difference between the obtained loss values ​​being less than the second threshold; this embodiment does not impose any limitations on this.

[0079] Alternatively, the process of determining the loss value can be expressed as: L = L v2t +L t2v L represents the loss value. B represents the quantity of sample text data, s(v i ,t i ,) represents sample text data information v i and sample text data information v i Corresponding sample image data information t i The third similarity information between them.

[0080] S203. Based on the first information and the image data information to be processed, determine the target image data information.

[0081] In this embodiment, the target image data information is the image data information retrieved from the image data information to be processed based on the first information. In this embodiment, the text data information to be processed and the image data information to be processed are processed to obtain the first information, and then the image data information to be processed is retrieved based on the first information, which can improve the accuracy and comprehensiveness of the video retrieval results.

[0082] In one specific embodiment, the step of determining the target image data information based on the first information and the image data information to be processed specifically includes: filtering the image data information to be processed based on the first information to obtain the target image data information.

[0083] In another specific implementation, refer to Figure 6As shown, in step S203 above, determining the target image data information based on the first information and the image data information to be processed may include steps S501 to S502, as follows:

[0084] S501. Based on the first information, determine the target information.

[0085] In this embodiment, the target information is determined based on the second similarity information between the first information and the first image segment information in the image data information to be processed. Video retrieval based on the target information can fully consider the temporal proximity and feature similarity between image segments, reduce the inclusion of irrelevant segments in the retrieval results, and improve the accuracy and comprehensiveness of the video retrieval results.

[0086] In one specific embodiment, the step of determining target information based on first information specifically includes: determining a second feature value corresponding to each first image segment based on second similarity information of the first image segment information in the image data information to be processed, wherein the second similarity information is used to characterize the degree of similarity between any two first image segment information; and calculating and processing the first feature value and the second feature value to obtain the target information.

[0087] Optionally, the step of calculating and processing the first feature value and the second feature value to obtain the target information specifically includes: performing a weighted summation of the first feature value and the second feature value to obtain the target information. The target information includes the target feature value corresponding to each first image segment, and the process of determining the target feature value can be expressed as follows: This represents the target feature value corresponding to the i-th first image segment. This represents the first feature value corresponding to the i-th first image segment information. Let α represent the second feature value corresponding to the i-th first image segment information, β represent the first weight information, and β represent the second weight information. α and β can be set according to actual needs.

[0088] In one specific embodiment, the step of determining the second feature value corresponding to each first image segment based on the second similarity information of the first image segment information in the image data information to be processed specifically includes: for any first image segment information in the image data information to be processed, determining the third image segment information corresponding to the first image segment information based on the second similarity information; and determining the second feature value corresponding to each first image segment information based on the third image segment information corresponding to each first image segment information.

[0089] In this embodiment, the third image fragment information is the image fragment information in the image data information to be processed that satisfies the first condition regarding the second similarity information between it and the first image fragment information. The second similarity information satisfying the first condition can be that the similarity value corresponding to the second similarity information is greater than the similarity threshold, or it can be that the similarity value corresponding to the second similarity information is within the similarity value range. This embodiment does not impose any limitations on this.

[0090] In one specific embodiment, when determining the third image segment information corresponding to the first image segment information based on the second similarity information, a graph model can be determined based on the second similarity information and the first image segment information. Then, the third image segment information corresponding to each first image segment information can be determined based on the graph model. Specifically, the graph model uses the first image segment information as nodes. When the second similarity between two nodes satisfies a first condition, an edge is created between the two nodes, and the nodes connected by this edge are the third image segment information corresponding to the first image segment information.

[0091] In one specific embodiment, the step of determining the second feature value corresponding to each first image segment based on the third image segment information corresponding to each first image segment specifically includes: determining a fourth feature value corresponding to each first image segment based on the third feature value corresponding to the third image segment information; updating the third feature value based on the fourth feature value, and continuing to execute the step of determining the fourth feature value corresponding to each first image segment based on the third feature value corresponding to the third image segment information, until the number of updates of the third feature value reaches a threshold; and determining the fourth feature value corresponding to each first image segment as the second feature value corresponding to each first image segment. This embodiment of the application determines the second feature value corresponding to each first image segment by iteratively updating the third feature value, which can fully consider the temporal adjacency and feature similarity relationships between image segments, thereby improving the accuracy of the determined second feature value.

[0092] Optionally, the third feature value is the initial feature value corresponding to each first image segment information. The third feature value can be set according to actual needs. For example, if the number of first image segment information is M, then the initial feature value corresponding to each first image segment information can be set to...

[0093] In a specific embodiment, the step of determining the fourth feature value corresponding to each first image segment based on the third feature value of the third image segment corresponding to each first image segment specifically includes: for any first image segment, calculating and processing the third feature value corresponding to the first image segment and the second similarity information between the first image segment and the corresponding third image segment to obtain the fifth feature value corresponding to the first image segment; calculating and processing the fifth feature value corresponding to the first image segment, the number of third image segments corresponding to the first image segment, and the first parameter information to obtain the sixth feature value corresponding to the first image segment; calculating and processing the number of first image segments and the first parameter information to obtain the seventh feature value; and calculating and processing the sixth and seventh feature values ​​to obtain the fourth feature value corresponding to each first image segment.

[0094] Alternatively, the process of determining the fourth eigenvalue can be expressed as: Among them, PR(V j ) represents the information of the first image segment V j The corresponding fourth feature value, d represents the first parameter, N represents the number of information in the first image segment, In(V j ) represents the information of the first image segment V j The corresponding set of third image fragment information, PR(V) k ) represents the information of the first image segment V j The corresponding third image fragment information V k The corresponding third eigenvalue, w(V) k V j ) represents the information of the first image segment V j and the third image fragment information V k The second similarity information between them, outdegree(V) k ) represents the information of the first image segment V j The number of corresponding third image fragments.

[0095] S502. Based on the target information, filter the image data information to be processed to obtain the target image data information.

[0096] In this embodiment, the image data information to be processed includes at least one first image segment information, and the target information includes a target feature value corresponding to each first image segment information. When filtering the image data information to be processed based on the target information, the first image segment information whose target feature value satisfies the third condition can be filtered from the image data information to be processed based on the target information, thereby obtaining the target image data information. Optionally, the target feature value satisfying the third condition can be that the target feature value is greater than a third threshold, or the target feature value satisfying the third condition can be that the target feature value is within the feature value range; this embodiment does not limit this.

[0097] In summary, the data processing method provided in this implementation scheme obtains text data and image data to be processed, processes them to obtain first information, and determines target image data based on the first information and the image data to be processed. This scheme improves the accuracy and comprehensiveness of video retrieval results by processing the text data and image data to be processed to obtain the first information and determining the target image data based on the first information and the image data to be processed. Furthermore, preprocessing the image data to be processed to obtain preprocessed image data, and then processing the text data and preprocessed image data to obtain the first information, reduces the computational load of data processing, extracts representative image features, and accurately extracts text semantic features, further improving the accuracy and comprehensiveness of video retrieval results. Even further, determining the target information based on the second similarity information between the first information and the first image segment information, and then filtering the image data to be processed based on the target information, fully considers the temporal adjacency and feature similarity relationships between image segments, reduces the inclusion of irrelevant segments in the retrieval results, and further improves the accuracy and comprehensiveness of video retrieval results.

[0098] To better implement the data processing method in the embodiments of this application, a data processing system is also provided in the embodiments of this application, such as... Figure 7 As shown, the data processing system 600 includes:

[0099] Information acquisition module 610 is used to acquire text data information to be processed and image data information to be processed;

[0100] Data processing module 620 is used to process the text data information and the image data information to be processed to obtain the first information;

[0101] The data determination module 630 is used to determine the target image data information based on the first information and the image data information to be processed.

[0102] In this embodiment, first information is obtained by processing the text data information and the image data information to be processed, and the target image data information is determined based on the first information and the image data information to be processed, which can improve the accuracy and comprehensiveness of video retrieval results.

[0103] In some embodiments of this application, the data processing module 620 processes the text data information to be processed and the image data information to be processed to obtain first information, including:

[0104] The image data to be processed is preprocessed to obtain preprocessed image data.

[0105] The text data to be processed and the preprocessed image data are processed to obtain the first information.

[0106] In some embodiments of this application, the image data information to be processed includes at least one first image segment information. The data processing module 620 preprocesses the image data information to be processed to obtain preprocessed image data information, including:

[0107] For any first image segment in the image data information to be processed, the first image segment information is filtered to obtain the first image data information corresponding to the first image segment information;

[0108] The first image data information is segmented to obtain the second image data information corresponding to the first image segment information; the second image data information includes at least one second image segment information.

[0109] The second image segment information in the second image data information is filtered to obtain preprocessed image data information.

[0110] In some embodiments of this application, the first information is obtained by processing the text data information to be processed and the preprocessed image data information through a target processing model. The target processing model includes: a first feature extraction module, a second feature extraction module, and an information determination module.

[0111] The input end of the first feature extraction module is configured to receive preprocessed image data information, the input end of the second feature extraction module is configured to receive text data information to be processed, and the output ends of the first feature extraction module and the second feature extraction module are respectively connected to the input end of the information determination module.

[0112] The first feature extraction module is configured to extract features from the preprocessed image data to obtain first feature information;

[0113] The second feature extraction module is configured to extract features from the text data to be processed, thereby obtaining second feature information.

[0114] The information determination module is configured to determine the first information based on the first feature information and the second feature information.

[0115] In some embodiments of this application, the first feature extraction module includes: a first feature extraction unit, a normalization unit, a feature mapping unit, and a pooling unit; wherein, the input end of the first feature extraction unit is configured to receive preprocessed image data information, the output end of the first feature extraction unit is connected to the input end of the normalization unit, the output end of the normalization unit is connected to the input end of the feature mapping unit, and the output end of the feature mapping unit is connected to the input end of the pooling unit.

[0116] The first feature extraction unit is configured to extract features from the preprocessed image data to obtain the third feature information;

[0117] The normalization unit is configured to normalize the third feature information to obtain the fourth feature information;

[0118] The feature mapping unit is configured to perform feature mapping processing on the fourth feature information to obtain the fifth feature information;

[0119] The pooling unit is configured to perform pooling processing on the fifth feature information to obtain the first feature information.

[0120] In some embodiments of this application, the image data information to be processed includes at least one first image segment information, the first feature information includes an image feature representation corresponding to each first image segment information, and the information determination module determines first information based on the first feature information and the second feature information, including:

[0121] The image feature representation and second feature information corresponding to each first image segment are calculated and processed to obtain the first similarity information corresponding to each first image segment.

[0122] The first similarity information is processed by information mapping to obtain the first information.

[0123] In some embodiments of this application, the information determination module calculates and processes the image feature representation and second feature information corresponding to each first image segment information to obtain first similarity information corresponding to each first image segment information, including:

[0124] The second feature information is transposed to obtain the sixth feature information;

[0125] The image feature representation and sixth feature information corresponding to each first image segment are calculated and processed to obtain the seventh feature information corresponding to each first image segment.

[0126] The image feature representation corresponding to each first image segment is calculated and processed to obtain the eighth feature information corresponding to each first image segment.

[0127] The second feature information is processed to obtain the ninth feature information;

[0128] The seventh, eighth, and ninth feature information are calculated and processed to obtain the first similarity information corresponding to each first image segment.

[0129] In some embodiments of this application, the data determination module 630 determines target image data information based on first information and image data information to be processed, including:

[0130] Based on the initial information, determine the target information;

[0131] The target image data is obtained by filtering the target information.

[0132] In some embodiments of this application, the image data information to be processed includes at least one first image segment information. The first information includes a first feature value corresponding to each first image segment information. The data determination module 630 determines target information based on the first information, including:

[0133] Based on the second similarity information of the first image segment information in the image data information to be processed, a second feature value corresponding to each first image segment information is determined. The second similarity information is used to characterize the degree of similarity between any two first image segment information.

[0134] The first and second eigenvalues ​​are calculated and processed to obtain the target information.

[0135] In some embodiments of this application, the data determination module 630 determines a second feature value corresponding to each first image segment based on second similarity information of the first image segment information in the image data information to be processed, including:

[0136] For any first image segment in the image data to be processed, a third image segment corresponding to the first image segment is determined based on the second similarity information; the third image segment is an image segment in the image data to be processed whose second similarity information with the first image segment satisfies a first condition;

[0137] Based on the third image segment information corresponding to each first image segment information, a second feature value corresponding to each first image segment information is determined.

[0138] In some embodiments of this application, the data determination module 630 determines a second feature value corresponding to each first image segment based on the third image segment information corresponding to each first image segment information, including:

[0139] Based on the third feature value of the third image segment information corresponding to each first image segment information, determine the fourth feature value corresponding to each first image segment information;

[0140] The third feature value is updated based on the fourth feature value, and the process of determining the fourth feature value corresponding to each first image segment information based on the third feature value corresponding to the third image segment information continues until the number of updates of the third feature value reaches the number threshold.

[0141] The fourth feature value corresponding to each first image segment information is determined as the second feature value corresponding to each first image segment information.

[0142] In some embodiments of this application, the data determination module 630 determines a fourth feature value corresponding to each first image segment based on a third feature value of the third image segment information corresponding to each first image segment information, including:

[0143] For any first image segment information, the third feature value corresponding to the first image segment information and the second similarity information between the first image segment information and the corresponding third image segment information are calculated and processed to obtain the fifth feature value corresponding to the first image segment information.

[0144] The fifth feature value corresponding to the first image segment information, the number of third image segment information corresponding to the first image segment information, and the first parameter information are calculated and processed to obtain the sixth feature value corresponding to the first image segment information.

[0145] The number of information segments and the first parameter information of the first image segment are calculated and processed to obtain the seventh feature value;

[0146] The sixth and seventh feature values ​​are calculated and processed to obtain the fourth feature value corresponding to each first image segment information.

[0147] In some embodiments of this application, the information acquisition module 610 acquires text data information to be processed and image data information to be processed, including:

[0148] Obtain the first text data information and the third image data information;

[0149] The third image data information is segmented to obtain the image data information to be processed;

[0150] The first text data and the third image data are fused to obtain the text data to be processed.

[0151] In some embodiments of this application, the information acquisition module 610 performs fusion processing on the first text data information and the third image data information to obtain the text data information to be processed, including:

[0152] Based on the feature representation of the third image data information, corresponding second text data information is generated, and the second text data information is used to characterize the information features of the third image data information.

[0153] Information recognition is performed on the first text data to obtain the first candidate text information;

[0154] Information recognition is performed on the second text data to obtain the second candidate text information;

[0155] The first and second candidate text information are fused to obtain the text data information to be processed.

[0156] This application embodiment also provides a computer device, the computer device including:

[0157] One or more processors;

[0158] Memory; and

[0159] One or more applications, wherein the applications are stored in memory and configured to be executed by a processor from the steps of the data processing method in any of the embodiments described above.

[0160] This application also provides a computer device, such as... Figure 8 As shown, it illustrates a structural schematic diagram of the computer device involved in the embodiments of this application, specifically:

[0161] The computer device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will understand that... Figure 8 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0162] The processor 801 is the control center of the computer device. It connects various parts of the computer device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, thereby providing overall monitoring of the computer device. Optionally, the processor 801 may include one or more processing cores; optionally, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 801.

[0163] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0164] The computer device also includes a power supply 803 that supplies power to the various components. Optionally, the power supply 803 can be logically connected to the processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 803 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0165] The computer device may also include an input unit 804, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0166] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 801 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 runs the application programs stored in the memory 802 to realize various functions, as follows:

[0167] Obtain the text data and image data to be processed;

[0168] The text data and image data to be processed are processed to obtain the first information;

[0169] Based on the first information and the image data to be processed, the target image data information is determined.

[0170] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0171] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the data processing methods provided in embodiments of this application. For example, the computer program loaded by the processor can execute the following steps:

[0172] Obtain the text data and image data to be processed;

[0173] The text data and image data to be processed are processed to obtain the first information;

[0174] Based on the first information and the image data to be processed, the target image data information is determined.

[0175] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0176] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0177] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0178] The above provides a detailed description of a data processing method and system for video retrieval provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method characterized by, include: Obtain the text data and image data to be processed; The text data and image data to be processed are processed to obtain the first information; Based on the first information and the image data information to be processed, the target image data information is determined.

2. The method of claim 1, wherein, The process of processing the text data information and the image data information to be processed to obtain the first information includes: The image data to be processed is preprocessed to obtain preprocessed image data. Based on the text data information to be processed and the preprocessed image data information, first information is obtained, which is used to characterize the degree of correlation between the image data information to be processed and the text data information to be processed.

3. The method of claim 2, wherein, The image data information to be processed includes at least one first image segment information; The preprocessing of the image data information to be processed to obtain preprocessed image data information includes: For any first image segment in the image data information to be processed, the first image segment is filtered to obtain the first image data information corresponding to the first image segment. The first image data information is segmented to obtain second image data information corresponding to the first image segment information; the second image data information includes at least one second image segment information. The second image segment information in the second image data information is filtered to obtain preprocessed image data information.

4. The method of claim 2, wherein, The first information is obtained by processing the text data information to be processed and the preprocessed image data information through a target processing model. The target processing model includes: a first feature extraction module, a second feature extraction module, and an information determination module. The input terminal of the first feature extraction module is configured to receive the preprocessed image data information, the input terminal of the second feature extraction module is configured to receive the text data information to be processed, and the output terminals of the first feature extraction module and the second feature extraction module are respectively connected to the input terminal of the information determination module. The first feature extraction module is configured to extract features from the preprocessed image data information to obtain first feature information; The second feature extraction module is configured to extract features from the text data to be processed to obtain second feature information; The information determination module is configured to determine first information based on the first feature information and the second feature information.

5. The method of claim 4, wherein, The first feature extraction module includes: a first feature extraction unit, a normalization unit, a feature mapping unit, and a pooling unit; wherein, the input end of the first feature extraction unit is configured to receive the preprocessed image data information, the output end of the first feature extraction unit is connected to the input end of the normalization unit, the output end of the normalization unit is connected to the input end of the feature mapping unit, and the output end of the feature mapping unit is connected to the input end of the pooling unit. The first feature extraction unit is configured to extract features from the preprocessed image data information to obtain third feature information; The normalization unit is configured to normalize the third feature information to obtain the fourth feature information; The feature mapping unit is configured to perform feature mapping processing on the fourth feature information to obtain the fifth feature information; The pooling unit is configured to perform pooling processing on the fifth feature information to obtain the first feature information.

6. The method of claim 4, wherein, The image data information to be processed includes at least one first image segment information, and the first feature information includes an image feature representation corresponding to each first image segment information; The step of determining the first information based on the first feature information and the second feature information includes: The image feature representation and the second feature information corresponding to each first image segment information are calculated and processed to obtain the first similarity information corresponding to each first image segment information; The first similarity information is processed by information mapping to obtain the first information.

7. The method of claim 6, wherein, The step of calculating and processing the image feature representation and the second feature information corresponding to each first image segment information to obtain the first similarity information corresponding to each first image segment information includes: The second feature information is transposed to obtain the sixth feature information; The image feature representation and the sixth feature information corresponding to each first image segment information are calculated and processed to obtain the seventh feature information corresponding to each first image segment information; The image feature representation corresponding to each first image segment is calculated and processed to obtain the eighth feature information corresponding to each first image segment. The second feature information is processed to obtain the ninth feature information; The seventh feature information, the eighth feature information, and the ninth feature information are calculated and processed to obtain the first similarity information corresponding to each first image segment information.

8. The method of claim 1, wherein, The step of determining the target image data information based on the first information and the image data information to be processed includes: Based on the first information, the target information is determined; Based on the target information, the image data information to be processed is filtered to obtain the target image data information.

9. The method of claim 8, wherein, The image data information to be processed includes at least one first image segment information, and the first information includes a first feature value corresponding to each first image segment information; The step of determining the target information based on the first information includes: Based on the second similarity information of the first image segment information in the image data information to be processed, a second feature value corresponding to each first image segment information is determined. The second similarity information is used to characterize the degree of similarity between any two first image segment information. The first feature value and the second feature value are calculated and processed to obtain the target information.

10. The method of claim 9, wherein, The step of determining the second feature value corresponding to each first image segment based on the second similarity information of the first image segment information in the image data information to be processed includes: For any first image segment in the image data information to be processed, a third image segment corresponding to the first image segment is determined based on the second similarity information; the third image segment is an image segment in the image data information to be processed that satisfies a first condition in terms of the second similarity information between it and the first image segment. Based on the third image fragment information corresponding to each of the first image fragment information, a second feature value corresponding to each of the first image fragment information is determined.

11. The method of claim 10, wherein, The step of determining the second feature value corresponding to each of the first image fragments based on the third image fragment information corresponding to each of the first image fragments includes: Based on the third feature value of the third image segment information corresponding to each first image segment information, a fourth feature value corresponding to each first image segment information is determined. The third feature value is updated based on the fourth feature value, and the step of determining the fourth feature value corresponding to each first image segment information based on the third feature value corresponding to each first image segment information continues to be executed until the number of updates of the third feature value reaches the number threshold. The fourth feature value corresponding to each of the first image fragment information is determined as the second feature value corresponding to each of the first image fragment information.

12. The method of claim 11, wherein, The step of determining the fourth feature value corresponding to each of the first image fragments based on the third feature value of the third image fragment information corresponding to each of the first image fragments includes: For any first image segment information, the third feature value corresponding to the first image segment information and the second similarity information between the first image segment information and the corresponding third image segment information are calculated and processed to obtain the fifth feature value corresponding to the first image segment information. The fifth feature value corresponding to the first image segment information, the number of third image segment information corresponding to the first image segment information, and the first parameter information are calculated and processed to obtain the sixth feature value corresponding to the first image segment information. The number of information segments and the first parameter information of the first image segment are calculated and processed to obtain the seventh feature value; The sixth and seventh feature values ​​are calculated and processed to obtain the fourth feature value corresponding to each first image segment information.

13. The method of claim 1, wherein, The acquisition of the text data information and image data information to be processed includes: Obtain the first text data information and the third image data information; The third image data information is segmented to obtain the image data information to be processed; The first text data information and the third image data information are fused to obtain the text data information to be processed.

14. The method of claim 13, wherein, The process of fusing the first text data information and the third image data information to obtain the text data information to be processed includes: Based on the feature representation of the third image data information, corresponding second text data information is generated, and the second text data information is used to characterize the information features of the third image data information. Information recognition is performed on the first text data information to obtain the first candidate text information; Information recognition is performed on the second text data information to obtain the second candidate text information; The first candidate text information and the second candidate text information are fused together to obtain the text data information to be processed.

15. A system, comprising: include: The information acquisition module is used to acquire text data and image data to be processed. The data processing module is used to process the text data information to be processed and the image data information to be processed to obtain the first information; The data determination module is used to determine the target image data information based on the first information and the image data information to be processed; Further, the data processing module processes the text data information to be processed and the image data information to be processed to obtain first information, including: The image data to be processed is preprocessed to obtain preprocessed image data. Based on the text data information to be processed and the preprocessed image data information, first information is obtained, which is used to characterize the degree of correlation between the image data information to be processed and the text data information to be processed; Further, the image data information to be processed includes at least one first image segment information, and the data processing module preprocesses the image data information to be processed to obtain preprocessed image data information, including: For any first image segment in the image data information to be processed, the first image segment is filtered to obtain the first image data information corresponding to the first image segment. The first image data information is segmented to obtain second image data information corresponding to the first image segment information; the second image data information includes at least one second image segment information. The second image segment information in the second image data information is filtered to obtain preprocessed image data information; Furthermore, the first information is obtained by processing the text data information to be processed and the preprocessed image data information through a target processing model, wherein the target processing model includes: a first feature extraction module, a second feature extraction module, and an information determination module; The input terminal of the first feature extraction module is configured to receive the preprocessed image data information, the input terminal of the second feature extraction module is configured to receive the text data information to be processed, and the output terminals of the first feature extraction module and the second feature extraction module are respectively connected to the input terminal of the information determination module. The first feature extraction module is configured to extract features from the preprocessed image data information to obtain first feature information; The second feature extraction module is configured to extract features from the text data to be processed to obtain second feature information; The information determination module is configured to determine the first information based on the first feature information and the second feature information; Further, the first feature extraction module includes: a first feature extraction unit, a normalization unit, a feature mapping unit, and a pooling unit; wherein, the input end of the first feature extraction unit is configured to receive the preprocessed image data information, the output end of the first feature extraction unit is connected to the input end of the normalization unit, the output end of the normalization unit is connected to the input end of the feature mapping unit, and the output end of the feature mapping unit is connected to the input end of the pooling unit; The first feature extraction unit is configured to extract features from the preprocessed image data information to obtain third feature information; The normalization unit is configured to normalize the third feature information to obtain the fourth feature information; The feature mapping unit is configured to perform feature mapping processing on the fourth feature information to obtain the fifth feature information; The pooling unit is configured to perform pooling processing on the fifth feature information to obtain the first feature information; Further, the image data information to be processed includes at least one first image segment information, the first feature information includes an image feature representation corresponding to each first image segment information, and the information determination module determines first information based on the first feature information and the second feature information, including: The image feature representation and the second feature information corresponding to each first image segment information are calculated and processed to obtain the first similarity information corresponding to each first image segment information; The first similarity information is processed by information mapping to obtain the first information; Further, the information determination module calculates and processes the image feature representation and the second feature information corresponding to each first image segment information to obtain first similarity information corresponding to each first image segment information, including: The second feature information is transposed to obtain the sixth feature information; The image feature representation and the sixth feature information corresponding to each first image segment information are calculated and processed to obtain the seventh feature information corresponding to each first image segment information; The image feature representation corresponding to each first image segment is calculated and processed to obtain the eighth feature information corresponding to each first image segment. The second feature information is processed to obtain the ninth feature information; The seventh feature information, the eighth feature information, and the ninth feature information are calculated and processed to obtain the first similarity information corresponding to each first image segment information; Further, the data determination module determines the target image data information based on the first information and the image data information to be processed, including: Based on the first information, the target information is determined; Based on the target information, the image data information to be processed is filtered to obtain the target image data information; Further, the image data information to be processed includes at least one first image segment information, the first information including a first feature value corresponding to each first image segment information, and the data determination module determines the target information based on the first information, including: Based on the second similarity information of the first image segment information in the image data information to be processed, a second feature value corresponding to each first image segment information is determined. The second similarity information is used to characterize the degree of similarity between any two first image segment information. The first feature value and the second feature value are calculated and processed to obtain the target information; Further, the data determination module determines a second feature value corresponding to each of the first image fragments based on the second similarity information of the first image fragment information in the image data information to be processed, including: For any first image segment in the image data information to be processed, a third image segment corresponding to the first image segment is determined based on the second similarity information; the third image segment is an image segment in the image data information to be processed that satisfies a first condition in terms of the second similarity information between it and the first image segment. Based on the third image segment information corresponding to each of the first image segment information, a second feature value corresponding to each of the first image segment information is determined; Further, the data determination module determines a second feature value corresponding to each of the first image segment information based on the third image segment information corresponding to each of the first image segment information, including: Based on the third feature value of the third image segment information corresponding to each first image segment information, a fourth feature value corresponding to each first image segment information is determined. The third feature value is updated based on the fourth feature value, and the step of determining the fourth feature value corresponding to each first image segment information based on the third feature value corresponding to each first image segment information continues to be executed until the number of updates of the third feature value reaches the number threshold. The fourth feature value corresponding to each of the first image fragment information is determined as the second feature value corresponding to each of the first image fragment information. Further, the data determination module determines a fourth feature value corresponding to each of the first image segment information based on the third feature value of the third image segment information corresponding to each of the first image segment information, including: For any first image segment information, the third feature value corresponding to the first image segment information and the second similarity information between the first image segment information and the corresponding third image segment information are calculated and processed to obtain the fifth feature value corresponding to the first image segment information. The fifth feature value corresponding to the first image segment information, the number of third image segment information corresponding to the first image segment information, and the first parameter information are calculated and processed to obtain the sixth feature value corresponding to the first image segment information. The number of information segments and the first parameter information of the first image segment are calculated and processed to obtain the seventh feature value; The sixth and seventh feature values ​​are calculated and processed to obtain the fourth feature value corresponding to each first image segment information; Furthermore, the information acquisition module acquires the text data information to be processed and the image data information to be processed, including: Obtain the first text data information and the third image data information; The third image data information is segmented to obtain the image data information to be processed; The first text data information and the third image data information are fused to obtain the text data information to be processed. Furthermore, the information acquisition module fuses the first text data information and the third image data information to obtain the text data information to be processed, including: Based on the feature representation of the third image data information, corresponding second text data information is generated, and the second text data information is used to characterize the information features of the third image data information. Information recognition is performed on the first text data information to obtain the first candidate text information; Information recognition is performed on the second text data information to obtain the second candidate text information; The first candidate text information and the second candidate text information are fused together to obtain the text data information to be processed.

16. An apparatus, comprising: The device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, It contains a computer program that is loaded by a processor to perform the steps of the method according to any one of claims 1 to 14.

Images