Data processing method, device, equipment, readable storage medium and program product

Abstract

Embodiments of the present application provide a data processing method, device and equipment, readable storage medium and program product. The method comprises: performing feature extraction processing on visual data to obtain a visual feature vector, and performing feature extraction processing on text description information to obtain a text feature vector; determining difference degree indication data between the visual feature vector and the text feature vector according to a difference degree calculation rule, and determining similarity degree indication data between the visual feature vector and the text feature vector according to a similarity degree calculation rule; and determining a consistency detection result of the visual data and the text description information according to the difference degree indication data and the similarity degree indication data. Through the embodiments of the present application, the accuracy of the image-text consistency detection can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to data processing methods, data processing apparatus, computer equipment, computer-readable storage media, and computer program products. Background Technology

[0002] With the rapid development of the internet, image (or video) and text data are widely disseminated on various online platforms. However, due to the diversity and complexity of information sources, inconsistencies between images and text frequently occur on the internet. This inconsistency not only leads to users receiving incorrect information and a poor user experience, but can also seriously impact data analysis and recommendation. Therefore, how to accurately perform image-text consistency detection is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, readable storage medium, and program product that can improve the accuracy of image-text consistency detection.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0005] Visual feature vectors are obtained by performing feature extraction on visual data, and text feature vectors are obtained by performing feature extraction on text description information.

[0006] The difference indicator data between visual feature vectors and text feature vectors is determined according to the difference calculation rules, and the similarity indicator data between visual feature vectors and text feature vectors is determined according to the similarity calculation rules.

[0007] Based on the difference indicator data and similarity indicator data, the consistency detection results between visual data and text description information are determined; the consistency detection results are used to indicate the degree of matching between visual data and text description information.

[0008] On the other hand, embodiments of this application provide a data processing apparatus, which includes:

[0009] The feature extraction unit is used to perform feature extraction processing on visual data to obtain visual feature vectors, and to perform feature extraction processing on text description information to obtain text feature vectors.

[0010] The processing unit is used to determine the difference indicator data between the visual feature vector and the text feature vector according to the difference calculation rules, and to determine the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules.

[0011] The processing unit is also used to determine the consistency detection result between visual data and text description information based on the difference indicator data and similarity indicator data; the consistency detection result is used to indicate the degree of matching between visual data and text description information.

[0012] In one possible implementation, the above processing unit is further used for:

[0013] In response to a search request from a first terminal device for search text, multiple candidate response data corresponding to the search text are determined from an information database; the multiple candidate response data include target response data, which includes visual data and text description information;

[0014] When the consistency detection result indicates that the visual data and text description information match, the target response data is returned to the first terminal device so that the target response data can be displayed on the first terminal device.

[0015] In one possible implementation, the processing unit described above, when determining the difference indication data between the visual feature vector and the text feature vector according to the difference calculation rules, specifically performs the following:

[0016] The visual feature vector is processed by vector mapping according to the first parameter matrix to obtain the first mapped vector;

[0017] The text feature vectors are processed by vector mapping according to the second parameter matrix to obtain the second mapped vector;

[0018] The first and second mapping vectors are processed by vector difference calculation to obtain the difference vector;

[0019] The difference vector is used to determine the difference indicator data between the visual feature vector and the text feature vector; wherein, the difference indicator data includes the classification prediction value obtained by calling the classifier to classify the difference vector.

[0020] In one possible implementation, the processing unit described above, when determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules, specifically performs the following:

[0021] The first and second mapping vectors are processed to calculate the similarity between the visual feature vector and the text feature vector, thus obtaining similarity indicator data between them.

[0022] In one possible implementation, the processing unit described above, when determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules, specifically performs the following:

[0023] The visual feature vector is processed by vector mapping according to the third parameter matrix to obtain the third mapping vector;

[0024] The text feature vectors are mapped according to the fourth parameter matrix to obtain the fourth mapping vector.

[0025] The third and fourth mapping vectors are processed to calculate the similarity between the visual feature vector and the text feature vector, thus obtaining similarity indicator data between them.

[0026] In one possible implementation, the processing unit, when determining the consistency detection result between visual data and textual description information based on the difference indicator data and similarity indicator data, specifically performs the following:

[0027] The difference indicator data and similarity indicator data are fused to obtain fused indicator data; the data fusion process is performed using the mean algorithm or a network layer.

[0028] Based on the fusion indicator data, determine the consistency detection results between visual data and text description information.

[0029] In one possible implementation, the difference indicator data and the similarity indicator data are determined by the target prediction model; the data processing device also includes a training unit, which is further used for:

[0030] Obtain sample data pairs; sample data pairs include sample visual data, sample text description information, and consistency labels;

[0031] The visual data of the samples is processed by feature extraction to obtain the visual feature vector of the samples, and the text description information of the samples is processed by feature extraction to obtain the text feature vector of the samples.

[0032] Based on the difference calculation rules, the sample difference indicator data between the sample visual feature vector and the sample text feature vector are determined, and based on the similarity calculation rules, the sample similarity indicator data between the sample visual feature vector and the sample text feature vector are determined.

[0033] The first difference data is determined based on the sample difference indicator data and the consistency label, and the second difference data is determined based on the sample similarity indicator data and the consistency label;

[0034] Based on the first and second difference data, the target difference data is determined, and the initial prediction model is tuned using the target difference data to obtain the target prediction model.

[0035] In one possible implementation, the training unit described above, when used to determine the target difference data based on the first difference data and the second difference data, specifically performs the following:

[0036] The first weighted difference data is determined based on the first difference data and the first weight parameter, and the second weighted difference data is determined based on the second difference data and the second weight parameter; wherein, the first difference data is calculated using the cross-entropy loss function based on the sample difference index data and the consistency label, and the second difference data is calculated using the mean squared error loss function based on the sample similarity index data and the consistency label;

[0037] The target difference data is obtained by summing the first weighted difference data and the second weighted difference data.

[0038] In one possible implementation, the above processing unit is further used for:

[0039] In response to the upload request from the second terminal device, the target data pair is parsed from the upload request; the target data pair includes visual data and text description information.

[0040] Based on the consistency detection results of visual data and text description information, determine the consistency label of the target data pair;

[0041] The target data pairs and their consistency tags are associated and stored in the information database;

[0042] The consistency label includes a first label for indicating data matching and a second label for indicating data mismatch; the consistency data label is used to determine the push priority based on the consistency label of the candidate response data during the process of pushing candidate response data to a third terminal device.

[0043] Accordingly, embodiments of this application provide a computer device, which includes:

[0044] A processor is a tool for implementing computer programs.

[0045] A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method.

[0046] Accordingly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0047] Accordingly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the data processing method described above.

[0048] This application embodiment performs feature extraction processing on visual data and text description information respectively to obtain visual feature vectors and text feature vectors; then, it determines the difference indication data between the visual feature vectors and the text feature vectors according to the difference calculation rules, and determines the similarity indication data between the visual feature vectors and the text feature vectors according to the similarity calculation rules, thereby improving the richness and diversity of reference information used for image-text consistency judgment; then, based on the difference indication data and similarity indication data, it determines the consistency detection result of visual data and text description information. Compared with using a single-dimensional measurement method, this can improve the accuracy of image-text consistency detection. Compared with manual image-text consistency judgment, it can achieve automated detection, reduce detection costs, and avoid the influence of human subjective factors on the judgment, thereby improving the accuracy of image-text consistency detection. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0052] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0053] Figure 4A This is a schematic diagram of a scenario where the text and images do not match, provided in an embodiment of this application.

[0054] Figure 4B This is a flowchart illustrating the calculation of target difference data provided in an embodiment of this application;

[0055] Figure 4C This is a flowchart of a target prediction model provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0059] This application involves various types of data, including visual data, text description information, and sample data pairs. When the above embodiments of this application are applied to specific products or technologies, the collection, use, and processing of related data should comply with relevant laws and regulations. Before collecting data, the information processing rules should be communicated and the individual consent of the recipient should be obtained. Related data should be processed in strict accordance with legal requirements and information processing rules, and technical measures should be taken to ensure the security of related data.

[0060] The data processing system provided in the embodiments of this application will now be described in conjunction with the accompanying drawings. The data processing system is suitable for implementing the data processing method provided in the embodiments of this application.

[0061] Please see Figure 1 This figure is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application. The data processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network (LAN), a wide area network (WAN), or the mobile internet.

[0062] Terminal equipment 101 may also be referred to as terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal equipment may be a smart home appliance, a handheld device with wireless communication capabilities (such as a smartphone or tablet), a computing device (such as a personal computer (PC)), an in-vehicle terminal, a smart voice interaction device, a wearable device, or other smart devices, but is not limited to these.

[0063] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0064] In one possible implementation, server 102 can perform feature extraction processing on visual data (e.g., images, videos, etc.) to obtain visual feature vectors, and perform feature extraction processing on textual description information (e.g., textual description information corresponding to images, videos, etc.) to obtain textual feature vectors. Server 102 determines difference indicator data between visual feature vectors and textual feature vectors according to difference calculation rules (difference indicator data is used to indicate the difference between visual feature vectors and textual feature vectors), and determines similarity indicator data between visual feature vectors and textual feature vectors according to similarity calculation rules (similarity indicator data is used to indicate the similarity between visual feature vectors and textual feature vectors). Server 102 jointly determines the consistency detection result between visual data and textual description information based on difference indicator data and similarity indicator data (consistency detection result is used to indicate the degree of matching between visual data and textual description information), thereby realizing image-text consistency detection. The above method does not rely on manual identification of whether visual data and text description information are consistent. Instead, it automatically and accurately judges whether visual data and text description information match and are consistent through image-text consistency detection, ensuring that image-text consistency detection is carried out efficiently. It can be well applied to content verification, information retrieval and multimedia data analysis, and helps to help judge the accuracy and content relevance of image and text information.

[0065] In one possible implementation, an object (such as a user) can perform video searches via terminal device 101 (such as a first terminal device), for example, by searching for short videos using search text. Based on this, terminal device 101 can send a search request to server 102 for the search text (the search request carries the search text, such as "Double Eleven Shopping Guide"). In response to the search request, server 102 determines multiple candidate response data corresponding to the search text from a database (for example, server 102 can perform similarity matching between the search text and the video titles of short videos to determine multiple candidate response data, such as short videos related to "Double Eleven Shopping Guide"). The multiple candidate response data includes target response data, which includes the aforementioned visual data and text description information. When the consistency detection results of the aforementioned visual data and text description information indicate a match, the target response data is returned to terminal device 101, allowing terminal device 101 to display the target response data to the object.

[0066] In other words, server 102 performs an image-text consistency check on each of the multiple candidate response data corresponding to the search text. Server 102 then returns the candidate response data that passes the image-text consistency check (i.e., image-text consistent) to terminal device 101. This method ensures that the search results returned to the user by the server are consistent with the images and text, reducing the possibility of users encountering irrelevant or misleading content during the search process and improving user experience. Furthermore, the automated image-text consistency check filters out inconsistent content, reducing the spread of misinformation and improving the platform's service quality and user satisfaction.

[0067] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. For example, the data processing method provided in the embodiments of this application can be executed not only by server 102, but also by other servers or server clusters that are different from server 102 and can communicate with terminal device 101 and / or server 102. Those skilled in the art will understand that... Figure 1 The number of terminal devices and servers shown is merely illustrative. Any number of terminal devices and servers can be configured according to business needs. Furthermore, as system architecture evolves and new business scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems. In subsequent embodiments, "terminal device" will refer to the aforementioned terminal device 101, and "server" will refer to the aforementioned server 102; further details will not be repeated in subsequent embodiments.

[0068] The data processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0069] This application provides a data processing method that can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 2 As shown, the data processing method may include, but is not limited to, the following steps S201-S203:

[0070] S201. Perform feature extraction processing on the visual data to obtain a visual feature vector, and perform feature extraction processing on the text description information to obtain a text feature vector.

[0071] In one possible implementation, visual data and textual description information can be a set of related data pairs. For example, visual data could refer to a video (or a video frame, or a video cover, etc.), and textual description information could refer to the video title (or the corresponding video description information). Alternatively, visual data could refer to an image, and textual description information could refer to the image's description. Computer equipment can perform image-text consistency checks on the related visual data and textual description information to obtain the consistency detection results for the related data pairs. By performing image-text consistency checks on the related visual data and textual description information, the matching degree and consistency of their content can be further verified, thereby verifying the accuracy of the information in the data pairs.

[0072] In one possible implementation, visual data and textual descriptions can be two unrelated sets of data. For example, visual data could refer to a video, and textual descriptions could refer to a search text (e.g., a text fragment used for data search tasks). Computer devices can then perform image-text consistency checks on unrelated visual data and textual descriptions to obtain a consistency result for the two sets of data. By performing image-text consistency checks on unrelated visual data and textual descriptions, the matching degree and consistency of the two sets of data in terms of content can be quickly and accurately determined, achieving automated data matching.

[0073] For ease of understanding, in the following embodiments, the example will be taken as visual data for the video and text description information for the corresponding video title. The following embodiments will not be described in detail again.

[0074] In one possible implementation, the computer device can extract key information from visual data (e.g., video) to obtain key information representing the video content. For example, the computer device can use keyframe extraction to obtain keyframes of the visual data to represent the visual modal information of the visual data. Here, the keyframe can be the video's cover frame, an intra-coded frame (I-frame), or other forms of image frames or image regions used to represent key information in the video; this application embodiment does not limit this. Furthermore, the computer device can perform image feature extraction on the keyframes to obtain visual feature vectors. Image feature extraction can be implemented using an image feature extraction network, such as VGG (Visual Geometry Group), ResNet (Deep Residual Networks), ResNeXt (Aggregated Residual Transformations for Deep Neural Networks), DenseNet (Densely Connected Convolutional Networks), SENet (Squeeze-and-Excitation Networks), etc.; this application embodiment does not limit this.

[0075] In one possible implementation, a computer device can perform text feature extraction (encoding) on ​​textual descriptive information (such as video titles) to obtain textual feature vectors that represent semantic information. Text feature extraction can be implemented using a text feature extraction network, such as BERT (Bidirectional Encoder Representations from Transformers), XL-Net (Extended Language Model), or ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately), etc. This application does not limit the specific implementation to these networks.

[0076] S202. Determine the difference indicator data between the visual feature vector and the text feature vector according to the difference calculation rules, and determine the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules.

[0077] In this embodiment, the computer device can predefine two rules for measuring the correlation between visual feature vectors and text feature vectors: a difference calculation rule and a similarity calculation rule. The difference calculation rule is a mathematical method for measuring the degree of difference between two feature vectors, while the similarity calculation rule is a mathematical method for measuring the degree of similarity between two feature vectors. Furthermore, the computer device can obtain difference indication data and similarity indication data between visual feature vectors and text feature vectors, improving the richness and diversity of reference information used for judging image-text consistency.

[0078] By using both difference and similarity indicators, the degree of matching (i.e., consistency) between visual data and textual descriptions can be more accurately assessed. These indicators are then used to determine the consistency detection results between the visual data and textual descriptions. Compared to using a single-dimensional measurement method, this improves the accuracy of image-text consistency detection. Furthermore, compared to manual image-text consistency judgment, it avoids the influence of subjective human factors, thus improving the accuracy of image-text consistency detection while also reducing detection costs.

[0079] In one possible implementation, the dissimilarity indicator data can be represented as a dissimilarity score, which is directly proportional to the degree of difference between the two vectors. Similarly, the similarity indicator data can also be represented as a similarity score, which is inversely proportional to the degree of difference between the two vectors.

[0080] S203. Based on the difference indicator data and similarity indicator data, determine the consistency detection results of visual data and text description information.

[0081] In this embodiment, the computer device determines the consistency detection result based on two results: difference indicator data and similarity indicator data. The consistency detection result indicates the degree of matching between visual data and textual description information. This method enables the computer device to comprehensively and accurately evaluate the consistency between visual data and textual description information from multiple dimensions. This avoids relying solely on a single discriminant dimension for consistency judgment, thereby improving the accuracy and reliability of image-text consistency detection.

[0082] Based on the above embodiments, the beneficial effects of this application are as follows: This application performs feature extraction processing on visual data and text description information respectively to obtain visual feature vectors and text feature vectors; then, it determines the difference indicator data between the visual feature vectors and the text feature vectors according to the difference calculation rules, and determines the similarity indicator data between the visual feature vectors and the text feature vectors according to the similarity calculation rules, thereby improving the richness and diversity of reference information used for image-text consistency judgment; furthermore, it determines the consistency detection result of visual data and text description information based on the difference indicator data and the similarity indicator data. Compared with using a single-dimensional measurement method, this can improve the accuracy of image-text consistency detection. Compared with manual image-text consistency judgment, it can achieve automated detection, reduce detection costs, and avoid the influence of human subjective factors on the judgment, thereby improving the accuracy of image-text consistency detection.

[0083] This application provides another data processing method, which can be executed by a computer device, such as a computer device that can be... Figure 1 Server 102 in the data processing system shown. For example... Figure 3 As shown, the data processing method may include, but is not limited to, the following steps S301-S306:

[0084] S301. In response to a search request from the first terminal device for the search text, determine multiple candidate response data corresponding to the search text from the information database; the multiple candidate response data includes target response data, and the target response data includes visual data and text description information.

[0085] In this embodiment, an object (such as a user) can perform video searches through a first terminal device, for example, by searching for short videos using search text. Therefore, the first terminal device can send a search request to a computer device for the search text (the search request carries the search text, such as "Double Eleven Shopping Guide"). Based on this, the computer device responds to the search request by determining multiple candidate response data corresponding to the search text from an information database (a database storing massive amounts of videos). For each candidate response data, the computer device can perform a text-image consistency check.

[0086] The above method performs an initial screening based on the search text to obtain multiple candidate response data, thereby enabling rapid retrieval of candidate data related to the search text and improving the system's response speed. Subsequently, the computer device performs an image-text consistency check on each of the multiple candidate response data (image-text consistency check can be seen as a secondary screening from the dimension of image-text consistency), thereby eliminating data with inconsistent images and text. The remaining data is then pushed to the target as search results, ensuring that the content seen by the user in the search results is highly relevant to their search text, and that each piece of content in the search results has image-text consistency. This improves the accuracy of information retrieval and user experience, increasing user trust and reliance on the system.

[0087] In subsequent steps S302-S306, the processing flow of image-text consistency detection will be explained using any one of the multiple candidate response data (denoted as target response data, which includes visual data and text description information) as an example.

[0088] In one possible implementation, the computer device determines multiple candidate response data corresponding to the search text from the information database. This can be achieved through any of the following methods: keyword matching (e.g., filtering relevant videos by matching keywords in the search text with video information in the information database, such as video titles, video descriptions, tags, etc.), semantic similarity calculation (e.g., using semantic similarity models such as BERT to calculate the semantic similarity between the search text and the video information in the information database), user-personalized recommendation (e.g., recommending videos that the user may be interested in by combining the user's search text, historical search records, and operational behavior), etc., which will not be elaborated here.

[0089] S302. Perform feature extraction processing on the visual data to obtain a visual feature vector, and perform feature extraction processing on the text description information to obtain a text feature vector.

[0090] For example, a computer device can use a ResNet50 model to perform offline extraction to obtain visual feature vectors corresponding to the visual data. These visual feature vectors can be denoted as E. img The calculation formula can be as follows:

[0091] E img =ResNet50(image)

[0092] Here, "image" represents a keyframe of visual data. A keyframe can be obtained by computer devices extracting key information from visual data (such as video). For example, a keyframe can be the cover frame of visual data (such as video).

[0093] For example, a computer device can use the BERT model to encode text description information to obtain a text feature vector corresponding to the text description information. The text feature vector can be denoted as E. text The calculation formula can be as follows:

[0094] E text =BERT(Title)

[0095] Here, Title represents the text description information corresponding to the visual data. For example, Title can be the video title of visual data (such as video).

[0096] It should be noted that the method for feature extraction processing of text description information in this application embodiment can also be replaced by other text encoding methods. At the same time, the method for feature extraction processing of visual data can also be replaced by other encoding methods, such as R-CNN (full name Region-CNN), fast-R-CNN, etc.

[0097] S303. Determine the difference indicator data between the visual feature vector and the text feature vector according to the difference calculation rules.

[0098] In this embodiment of the application, when processing multimodal data (such as images and text), different modalities typically have different feature representations. For example, image features are usually high-dimensional visual features, while text features are high-level semantic features of different dimensions. Therefore, when calculating the difference between visual feature vectors and text feature vectors, they need to be transformed into the same feature space (that is, uniformly represented as vectors of the same dimension) to facilitate subsequent calculations and analysis.

[0099] In one possible implementation, for step S303 above, one implementation can be as follows: (1)-(4):

[0100] (1) Perform vector mapping on the visual feature vector according to the first parameter matrix to obtain the first mapping vector.

[0101] (2) Perform vector mapping on the text feature vector according to the second parameter matrix to obtain the second mapping vector.

[0102] In this embodiment, vector mapping refers to the operation of transforming a vector into another vector space through a certain mapping function. This mapping can be linear (such as matrix multiplication) or non-linear (such as neural network layers). In this embodiment, the computer device uses different parameter matrices to perform vector mapping on visual feature vectors and text feature vectors respectively, to obtain a first mapped vector and a second mapped vector, so as to facilitate subsequent calculation of vector differences.

[0103] For example, a computer device can use matrix multiplication to perform vector mapping through a parameter matrix. The first mapped vector can be denoted as e1, and the second mapped vector can be denoted as e2. The calculation formula is as follows:

[0104] e1 = W1 × E img

[0105] e2=W2×E text

[0106] Wherein, W1 and W2 are the parameter matrices used for the visual feature vector (i.e., the first parameter matrix) and the parameter matrices used for the text feature vector (i.e., the second parameter matrix) respectively during the difference calculation process.

[0107] (3) Perform vector difference calculation on the first mapping vector and the second mapping vector to obtain the difference vector.

[0108] For example, the difference vector can be denoted as diff. e The calculation formula can be as follows:

[0109] diff e =e1-e2

[0110] (4) Determine the difference indicator data between the visual feature vector and the text feature vector based on the difference vector.

[0111] In this embodiment, the difference vector quantifies the difference between the visual feature vector and the text feature vector. Therefore, the computer device can determine the difference degree indication data based on the difference vector. In one possible implementation, the difference degree indication data includes the classification prediction value obtained by calling a classifier to classify the difference vector. That is, the computer device can use network layers (such as fully connected layers, multilayer perceptrons, classifiers, etc.) to classify the difference vector and obtain the classification prediction value. The classification prediction value can map the difference vector to a specific difference degree category or value, thereby providing a more intuitive difference degree indication. In one case, the difference degree indication data may refer to the classification prediction value.

[0112] For example, a computer device can input the difference vector into a discriminant layer, which performs feature classification based on a fully connected network and outputs a classification prediction value, which can be denoted as p1. The calculation process can be as follows:

[0113] p1 = Classifier(diff) e )

[0114] Here, "Classifier" refers to the discriminant layer, which can be a classifier. The lower the consistency between the text and images in a set of image-text data, the worse the diff. eThe larger the value, the larger p1 is; conversely, the higher the consistency between the image and text, the greater the diff. e The smaller the value, the smaller p1 is.

[0115] S304. Determine the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules.

[0116] In this embodiment of the application, when the computer device calculates the similarity between visual feature vectors and text feature vectors, it also needs to convert them into the same feature space (that is, uniformly represented as vectors of the same dimension) to facilitate subsequent calculation and analysis.

[0117] In one possible implementation, the computer device can determine how to calculate the similarity indicator data based on a vector mapping processing strategy. The vector mapping processing strategy refers to the strategy or rule used during vector mapping processing; the choice of the vector mapping processing strategy affects the final mapped vector result and subsequent calculation results based on the mapped vector result. In one possible implementation, the vector mapping processing strategy can include at least two cases: using parameter sharing rules and not using parameter sharing rules.

[0118] If a parameter-sharing rule is adopted, then when the computer device performs similarity and difference calculations, it will use the same parameter matrix for mapping the visual feature vectors, that is, perform shared matrix mapping, such as using parameter matrix W1. Simultaneously, the computer device will use the same parameter matrix for mapping the text feature vectors, such as using parameter matrix W2.

[0119] In one possible implementation, when the vector mapping processing strategy indicates the use of a parameter sharing rule, then, for step S304 above, one implementation can be as follows: perform vector similarity calculation on the first and second mapping vectors to obtain similarity indicator data between the visual feature vector and the text feature vector. This method allows feature learning on both the visual and text feature sides to mutually promote (encourage the model to learn the commonalities between the two modalities) and also mutually constrain each other. Simultaneously, it simplifies the model structure, reduces the number of parameters, and thus improves computational efficiency.

[0120] In this embodiment, the computer device can perform vector similarity calculation on the first mapping vector and the second mapping vector to obtain similarity indicator data. By calculating the similarity indicator data, the degree of similarity between visual features and text features can be quantified, thereby providing a basis for subsequent analysis and decision-making. Vector similarity calculation can employ, for example, cosine similarity, Euclidean distance, Manhattan distance, Pearson correlation coefficient, Mahalanobis distance, Jaccard similarity coefficient, etc.

[0121] For example, taking cosine similarity as an example, a computer device can calculate the cosine similarity between the first mapping vector and the second mapping vector. Then, under the parameter sharing rule, the similarity indicator data can also be denoted as p2, and the calculation process can be as follows:

[0122]

[0123] In the above formula, p2 can also be denoted as Cos_score, which indicates that the similarity indicator data is calculated based on cosine similarity. The lower the consistency between the text and images in a set of text and image data, the smaller p2 is; conversely, the higher the consistency between text and images, the larger p2 is.

[0124] Without parameter sharing, the computer would use different parameter matrices to map visual feature vectors during similarity and difference calculations. For example, parameter matrices W1 and W2 would be used for visual feature vectors, respectively, and W3 and W4 for text feature vectors, respectively, during similarity and difference calculations. This method allows feature learning on both the visual and textual sides to be performed separately, enabling the model to be finely adjusted for each modality, thereby improving the accuracy of the mapped vectors.

[0125] It should be noted that the vector mapping processing strategy can be obtained by the computer device before calculating the difference indicator data or before calculating the similarity indicator data; this application embodiment does not limit this.

[0126] In one possible implementation, when the vector mapping processing strategy indicates that the parameter sharing rule is not used, then, for step S304 above, one implementation can be as follows: (1)-(3):

[0127] (1) Perform vector mapping on the visual feature vector according to the third parameter matrix to obtain the third mapping vector.

[0128] (2) Perform vector mapping on the text feature vector according to the fourth parameter matrix to obtain the fourth mapping vector.

[0129] For example, when parameter sharing rules are not used, computer devices can use matrix multiplication to achieve vector mapping through parameter matrices. The third mapped vector can be denoted as e3, and the fourth mapped vector can be denoted as e4. The calculation formula is as follows:

[0130] e3 = W3 × E img

[0131] e4 = W4 × E text

[0132] Wherein, W3 and W4 are the parameter matrices used for the visual feature vector (i.e., the third parameter matrix) and the parameter matrix used for the text feature vector (i.e., the fourth parameter matrix) respectively during the similarity calculation process when the parameter sharing rule is not adopted. It should be noted that W1, W2, W3, and W4 in this embodiment can be randomly initialized parameter matrices, and the parameter values ​​of the parameter matrices will be gradually optimized as the model training iterates.

[0133] (3) Perform vector similarity calculation on the third and fourth mapping vectors to obtain similarity indicator data between visual feature vectors and text feature vectors.

[0134] In this embodiment of the application, the computer device can perform vector similarity calculation on the third mapping vector and the fourth mapping vector to obtain similarity indicator data. By calculating the similarity indicator data, the degree of similarity between visual features and text features can be quantified, thereby providing a basis for subsequent analysis and decision-making.

[0135] For example, taking cosine similarity as an example, a computer device can calculate the cosine similarity between the third and fourth mapping vectors. Then, the similarity indicator data can be denoted as p2, and the calculation process is as follows:

[0136]

[0137] In one possible implementation, the computer device may also follow a preset business logic, without determining the vector mapping processing strategy, and instead, after obtaining the difference indication data, directly perform vector similarity calculation on the first and second mapping vectors to obtain the similarity indication data between the visual feature vector and the text feature vector. This will not be elaborated further here.

[0138] S305. Based on the difference indicator data and similarity indicator data, determine the consistency detection results of visual data and text description information.

[0139] In one possible implementation, for step S305 above, one implementation can be as follows: (1)-(2):

[0140] (1) Perform data fusion processing on the difference indicator data and the similarity indicator data to obtain fused indicator data.

[0141] In this embodiment, image consistency judgment can be viewed as a comparison and alignment of modal vectors on both sides. The comparison relationship between the two modal vectors is characterized by introducing difference indicator data and similarity indicator data. Based on this, data fusion processing can integrate data from different sources or different modalities to obtain a comprehensive and more complete data representation, thereby improving the overall model's performance and robustness. Data fusion processing can be performed using a mean algorithm or a network layer.

[0142] For example, let the fused indicator data be denoted as P. If the mean algorithm is used for processing, the formula for calculating the fused indicator data can be as follows:

[0143]

[0144] It should be noted that, considering that the higher the consistency between the text and images in a set of image and text data, the smaller the difference indicator p1 and the larger the similarity indicator p2, the formula for calculating the fusion indicator data can also be as follows:

[0145]

[0146] Specifically, (1-p1) can transform the difference indicator data into reference data for indicating the consistency between the image and text. The average of this reference data and the similarity indicator data p2 is then used as the fusion indicator data, thus allowing the degree of image-text consistency to be indicated through the fusion indicator data. Besides this, computer devices can also use other methods that fuse two types of indicator data to determine the fusion indicator data, which will not be elaborated here.

[0147] If a network layer is used for processing, the network layer can integrate information from different sources by learning weights, thereby achieving the fusion of two types of indicator data. The processing can be represented as P = FC(p1, p2), where FC represents the processing operation of the network layer (such as a fully connected layer).

[0148] (2) Determine the consistency detection results of visual data and text description information based on the fusion indication data.

[0149] In this embodiment, the computer device determines the consistency detection result based on the fusion indication data, thereby fully considering the differences and similarities between visual data and text description information, strengthening the interaction between the two sides of information, avoiding noise and errors that may exist in a single data source, and thus improving the accuracy of image-text consistency detection.

[0150] In one possible implementation, the computer device can determine the consistency detection result based on the fused indication data using methods such as thresholding or classifiers. For example, if a thresholding method is used, a judgment threshold can be preset. If the fused indication data is greater than or equal to the judgment threshold, the visual data and text description information are considered consistent; if the fused indication data is less than the judgment threshold, the visual data and text description information are considered inconsistent. If a classifier is used, the consistency detection result can be further determined through subsequent network layers (such as fully connected layers, softmax layers, or other classifiers), which will not be elaborated here.

[0151] S306. When the consistency detection result indicates that the visual data and the text description information match, the target response data is returned to the first terminal device so that the target response data can be displayed on the first terminal device.

[0152] In this embodiment, when the visual data and text description information in the target response data are consistent (image and text are identical), then for the user, this is data that is relevant to the search text and of high quality (high quality is manifested in the consistency between image and text). In this case, the computer device can return the target response data to the first terminal device. For each candidate response data item among multiple candidate response data items, the computer device performs an image-text consistency check and returns the candidate response data that passes the check to the first terminal device. The above method achieves fast and accurate feedback and response in user search scenarios, effectively eliminating irrelevant or low-quality data, preventing users from receiving irrelevant or misleading information, ensuring the overall quality of the returned candidate response data, and improving the user experience.

[0153] The computer device, through the above method, performs image-text consistency checks on each candidate response data after determining multiple candidate response data based on the search text, and returns the image-text consistent candidate response data to the user. In another possible implementation, the computer device may not need to perform image-text consistency checks on each candidate response data after determining multiple candidate response data based on the search text, but instead determine whether the data has image-text consistency when the user uploads the data, and upload the corresponding consistency tag for associated storage. Based on this, the computer device can also perform the following steps (1)-(3):

[0154] (1) In response to the upload request from the second terminal device, the target data pair is parsed from the upload request; the target data pair includes visual data and text description information.

[0155] (2) Determine the consistency label of the target data pair based on the consistency detection results of visual data and text description information.

[0156] In this embodiment, a user can upload a target data pair (e.g., the visual data of the target data pair may be a video, and the text description information may be the video title corresponding to the video) through a second terminal device. The computer device can then perform image-text consistency detection on the target data pair to obtain a consistency detection result. The method for determining the consistency detection result can be referred to the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0157] The computer equipment then determines the consistency label of the target data pair based on the consistency detection results. The consistency label is used to indicate whether the data matches. Specifically, the consistency label can include at least: a first label to indicate that the data matches (is consistent) and a second label to indicate that the data does not match (is inconsistent). By generating consistency labels for the target data pair, a basis for prioritizing subsequent data pushes is provided, thereby ensuring that the pushed data is high-quality data with consistent text and images.

[0158] (3) Associate the target data pair and the consistency tag of the target data pair and store them in the information database.

[0159] In this embodiment of the application, the computer device can associate and store the target data pair and its corresponding consistency tag in the information database for easy retrieval and use later.

[0160] In this embodiment, the consistency data tag is used to determine the push priority based on the consistency tag of the candidate response data during the process of pushing candidate response data to a third terminal device. That is, when pushing candidate response data to a third terminal device, the server can combine the consistency tag corresponding to the candidate response data to determine the push priority. The higher the push priority of candidate response data whose consistency tag indicates that the data matches (such as data with the first consistency tag), the lower the push priority of candidate response data whose consistency tag indicates that the data does not match (such as data with the second consistency tag).

[0161] The above method enables the computer device to associate and store consistency tags of uploaded data in a database at each stage of user data upload. Then, in the subsequent data recommendation stage (which could be recommending data to the user who uploaded the data or to other users), the computer device can determine multiple candidate response data from the database based on the user's search text, and then determine the corresponding priorities based on the consistency tags associated with these candidate response data. This provides a basis for data recommendation, ensuring that the pushed data is high-quality and consistent with the text and images.

[0162] In one possible implementation, after the computer device determines multiple candidate response data based on the search text, if there is a first candidate response data among the multiple candidate response data that does not have an associated consistency label, then the first candidate response data is subjected to image and text consistency detection to obtain the consistency detection result of the first candidate response data; based on the consistency detection result of the first candidate response data and the consistency labels of the candidate response data other than the first candidate response data among the multiple candidate response data, the push priority is jointly determined, which will not be elaborated here.

[0163] In this embodiment, the difference indicator data and similarity indicator data are determined by the target prediction model. Based on this, the training process of the target prediction model will be described below:

[0164] In one possible implementation, the training of the target prediction model can be carried out through the following steps (1)-(6):

[0165] (1) Obtain sample data pairs; sample data pairs include sample visual data, sample text description information, and consistency labels.

[0166] In this embodiment, a sample data pair refers to training samples used to train the target prediction model. A sample data pair includes sample visual data, corresponding sample text description information, and a consistency label. The consistency label serves as supervision information, indicating whether the sample visual data and sample text description information match, i.e., whether they have image-text consistency. For example, if the sample data pair has image-text consistency, the consistency label can be set to 1; if the sample data pair does not have image-text consistency, the consistency label can be set to 0.

[0167] (2) Perform feature extraction processing on the sample visual data to obtain the sample visual feature vector, and perform feature extraction processing on the sample text description information to obtain the sample text feature vector.

[0168] The specific implementation of step (2) can be found in the descriptions of the processing of visual data and text description information in the aforementioned embodiments, and will not be repeated here.

[0169] (3) Determine the sample difference indicator data between the sample visual feature vector and the sample text feature vector according to the difference calculation rules, and determine the sample similarity indicator data between the sample visual feature vector and the sample text feature vector according to the similarity calculation rules.

[0170] The specific implementation of step (3) can be found in the descriptions of the processing of visual feature vectors and text feature vectors in the aforementioned embodiments, and will not be repeated here.

[0171] (4) Determine the first difference data based on the sample difference indicator data and the consistency label, and determine the second difference data based on the sample similarity indicator data and the consistency label.

[0172] In one possible implementation, the computer device can calculate first difference data, such as a first loss function, using sample difference indicator data and consistency labels, and calculate second difference data, such as a second loss function, using sample similarity indicator data and consistency labels.

[0173] It should be noted that when determining the first difference data, the consistency label used is 0 to indicate that there is no difference between the image and text, and 1 to indicate that there is a difference between the image and text. However, when determining the second difference data, the consistency labels are reversed; that is, in this case, 1 indicates that there is no difference between the image and text, and 0 indicates that there is a difference between the image and text.

[0174] In one possible implementation, the first difference data can be calculated using the cross-entropy loss function based on the sample difference indicator data and the consistency label.

[0175] For example, the first difference data can be denoted as loss. cls The calculation formula can be as follows:

[0176]

[0177] Wherein, the first difference data represents the total loss function of the difference between multiple sets of sample data pairs, a i y represents the sample difference indicator data of the i-th sample data pair in a set of multiple sample data pairs. i The consistency label (i.e., y) represents the data pair of the i-th sample group. i =0 indicates that there is no difference between the image and the text, y i =1 indicates that there is a difference between the text and the image.

[0178] In one possible implementation, the second difference data can be calculated using the mean squared error loss function based on the sample similarity indicator data and the consistency label.

[0179] For example, the second difference data can be denoted as loss. regression The calculation formula can be as follows:

[0180]

[0181] Wherein, the second difference data represents the total similarity loss function of multiple sets of sample data pairs, b i This represents the sample similarity indicator data for the i-th sample data pair in a set of multiple sample data pairs. The consistency label for the i-th sample data pair (i.e., yi = 0 indicates that there is a difference between the image and the text) (This indicates that there is no difference between the text and the image).

[0182] (5) Determine the target difference data based on the first difference data and the second difference data.

[0183] In this embodiment of the application, the computer device can calculate the target difference data using the first difference data and the second difference data. For example, it can perform summation processing on the first difference data and the second difference data, weighted summation processing, etc., to obtain the total loss function used to train the initial model.

[0184] In one possible implementation, for step (5) above, i.e., determining the target difference data based on the first difference data and the second difference data, one implementation can be as follows: (1)-(2):

[0185] (1) Determine the first weighted difference data based on the first difference data and the first weight parameter, and determine the second weighted difference data based on the second difference data and the second weight parameter.

[0186] (2) The first weighted difference data and the second weighted difference data are summed to obtain the target difference data.

[0187] In steps (1)-(2) above, the computer device can flexibly adjust the relative importance of the first and second difference data in the target difference data by introducing weight parameters for the first and second difference data. Different weight parameters can reflect different optimization objectives or data characteristics, increasing the flexibility of the model. By summing the first weighted difference data and the second weighted difference data, the target difference data is obtained to improve the model training effect, enabling the model to comprehensively consider the influence of difference and similarity.

[0188] For example, the target variance data can be denoted as LOSS, and the calculation formula can be as follows:

[0189] LOSS = λ1loss cls +λ2loss regression

[0190] Where, loss cls Let λ represent the first difference data, λ1 represent the first weight parameter, and loss be... regression λ represents the second difference data, and λ2 represents the second weighting parameter.

[0191] (6) Use the target difference data to adjust the parameters of the initial prediction model to obtain the target prediction model.

[0192] In this embodiment, the computer device can use target difference data to perform parameter tuning (including optimization algorithms) on the initial prediction model to obtain the target prediction model. Through the above-mentioned supervised training method, the model can learn the difference and similarity features between the visual data and textual description information of the samples in the sample data pair, so as to improve the model's prediction accuracy and enhance its robustness and generalization ability. In this embodiment, during the training of the target prediction model, the computer device can simultaneously train the parameter matrix required for vector mapping processing. As the model training iterates, the parameter values ​​of the parameter matrix will be gradually optimized to achieve parameter sharing and collaborative optimization.

[0193] It should be noted that the computer equipment can also use multiple sets of sample data pairs to train the initial prediction model multiple times. When the training stopping condition is met, the target prediction model is obtained. The training stopping condition can be: reaching a preset number of training rounds or the current model's prediction accuracy reaching a preset requirement. Through this method, the target prediction model can more accurately capture the matching relationship between visual data and textual description information, thereby improving prediction accuracy.

[0194] The data processing method provided in this application can be well applied to various multi-dimensional feature interaction scenarios, such as news and video-based text-image interaction scenarios. Taking the video search scenario in a video search engine as an example, the query results of a video search engine include multiple sets of data, each set consisting of a video and a video title. However, there may be inconsistencies between the video and the video title, i.e., the video content does not match the video title.

[0195] like Figure 4A As shown, Figure 4AThis is an exemplary embodiment of the present application illustrating a scenario of mismatched text and image content. For example, in a set of data, a video is related to game A, but the video title is related to game B. The reasons for this inconsistency can include at least the following: First, an anomaly occurs during the parsing of webpage data to obtain the video and video title when obtaining webpage data through methods such as web crawling; second, a user uploads a video title that does not match the video content, such as due to an upload error or an attempt to increase the video's appeal.

[0196] To address the aforementioned issues, a target prediction model (such as a text-to-image consistency detection model) can be used to perform consistency detection on videos and video titles. Before training the target prediction model, multiple sets of sample data pairs need to be prepared in advance. Each set of sample data pairs includes sample text description information, sample visual data, and consistency labels. Specifically, this can refer to: video title, video keyframes (e.g., video cover), and corresponding consistency labels. The data format of the sample data pairs can be shown in Table 1 below:

[0197] Table 1

[0198] Video title Video keyframes Consistency Labels Tips for ranking up Image 1 1 Game A's closed beta test has begun, come and experience it! Image 2 0 The new character is too weak; their damage is too low. Image 3 1 The pace of mobile phone iteration is too fast Image 4 1 … … …

[0199] The training process of the target prediction model will be explained below: Figure 4B As shown, Figure 4B This is a flowchart illustrating the calculation of target difference data according to an exemplary embodiment of this application. The main processing steps are as follows:

[0200] 1. Perform feature extraction processing on the sample visual data to obtain the sample visual feature vector, and perform feature extraction processing on the sample text description information to obtain the sample text feature vector.

[0201] 2. Determine the sample difference indicator data between the sample visual feature vector and the sample text feature vector according to the difference calculation rules.

[0202] For example, a computer device can perform vector mapping processing on the sample visual feature vector and the sample text feature vector respectively to obtain a first mapping vector and a second mapping vector. Then, it can calculate the difference vector between the first mapping vector and the second mapping vector, process the difference vector through a network layer, and input the difference vector into a classifier (such as an FNN model) to obtain sample difference indication data.

[0203] 3. Determine the sample similarity indicator data between the sample visual feature vector and the sample text feature vector according to the similarity calculation rules.

[0204] For example, a computer device can perform vector similarity calculation on the first mapping vector and the second mapping vector to obtain similarity indicator data.

[0205] 4. Calculate the target difference data based on the sample difference indicator data, similarity indicator data, and consistency label.

[0206] For example, a computer device can calculate first difference data based on sample difference indicator data and consistency label, and determine second difference data based on sample similarity indicator data and consistency label, and then determine target difference data based on the first difference data and the second difference data.

[0207] After obtaining the target difference data, the computer device can use the target difference data to perform parameter tuning on the initial prediction model to obtain the target prediction model. The specific implementation methods of the above steps can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0208] The following will explain the process of using the target prediction model: Figure 4C As shown, Figure 4C This is a flowchart illustrating the processing of a target prediction model provided in an exemplary embodiment of this application. The main processing steps are as follows:

[0209] 1. Perform feature extraction processing on visual data to obtain visual feature vectors, and perform feature extraction processing on text description information to obtain text feature vectors.

[0210] 2. Determine the difference indicator data between the visual feature vector and the text feature vector according to the difference calculation rules.

[0211] 3. Determine the similarity indicator data between visual feature vectors and text feature vectors according to the similarity calculation rules.

[0212] 4. Perform data fusion processing on the difference indicator data and similarity indicator data to obtain fused indicator data.

[0213] 5. Based on the fusion indicator data, determine the consistency detection results between visual data and text description information.

[0214] The specific implementation methods of the above steps can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0215] This application's embodiments construct an object detection model for image-text consistency detection tasks by using feature differences and feature similarities between sample data pairs, effectively improving the depth of feature interaction. Furthermore, by introducing a shared matrix mapping (i.e., parameter sharing rules), feature learning on both the visual and textual sides mutually promotes and constrains each other, while also simplifying the model structure and reducing the number of parameters.

[0216] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0217] Please see Figure 5 This figure is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. This data processing device can be installed in the computer equipment provided in the embodiment of this application, and the computer equipment can be as described above. Figure 1 Server 102 in the data processing system shown. Figure 5 The data processing device shown can be a computer program running on a computer device, which can be used to execute... Figure 2 or Figure 3 Some or all of the steps in the method embodiments shown. Please refer to [link / reference]. Figure 5 The data processing apparatus may include the following units:

[0218] The feature extraction unit 501 is used to perform feature extraction processing on visual data to obtain visual feature vectors, and to perform feature extraction processing on text description information to obtain text feature vectors.

[0219] The processing unit 502 is used to determine the difference indication data between the visual feature vector and the text feature vector according to the difference calculation rules, and to determine the similarity indication data between the visual feature vector and the text feature vector according to the similarity calculation rules.

[0220] The processing unit 502 is further configured to determine the consistency detection result between visual data and text description information based on the difference indicator data and the similarity indicator data; the consistency detection result is used to indicate the degree of matching between visual data and text description information.

[0221] In one possible implementation, the processing unit 502 is further configured to:

[0222] In response to a search request from a first terminal device for search text, multiple candidate response data corresponding to the search text are determined from an information database; the multiple candidate response data include target response data, which includes visual data and text description information;

[0223] When the consistency detection result indicates that the visual data and text description information match, the target response data is returned to the first terminal device so that the target response data can be displayed on the first terminal device.

[0224] In one possible implementation, the processing unit 502, when determining the difference indication data between the visual feature vector and the text feature vector according to the difference calculation rules, specifically performs the following:

[0225] The visual feature vector is processed by vector mapping according to the first parameter matrix to obtain the first mapped vector;

[0226] The text feature vectors are processed by vector mapping according to the second parameter matrix to obtain the second mapped vector;

[0227] The first and second mapping vectors are processed by vector difference calculation to obtain the difference vector;

[0228] The difference vector is used to determine the difference indicator data between the visual feature vector and the text feature vector; wherein, the difference indicator data includes the classification prediction value obtained by calling the classifier to classify the difference vector.

[0229] In one possible implementation, the processing unit 502, when determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules, specifically performs the following:

[0230] The first and second mapping vectors are processed to calculate the similarity between the visual feature vector and the text feature vector, thus obtaining similarity indicator data between them.

[0231] In one possible implementation, the processing unit 502, when determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules, specifically performs the following:

[0232] The visual feature vector is processed by vector mapping according to the third parameter matrix to obtain the third mapping vector;

[0233] The text feature vectors are mapped according to the fourth parameter matrix to obtain the fourth mapping vector.

[0234] The third and fourth mapping vectors are processed to calculate the similarity between the visual feature vector and the text feature vector, thus obtaining similarity indicator data between them.

[0235] In one possible implementation, the processing unit 502, when determining the consistency detection result between visual data and textual description information based on the difference indicator data and similarity indicator data, specifically performs the following:

[0236] The difference indicator data and similarity indicator data are fused to obtain fused indicator data; the data fusion process is performed using the mean algorithm or a network layer.

[0237] Based on the fusion indicator data, determine the consistency detection results between visual data and text description information.

[0238] In one possible implementation, the difference indicator data and the similarity indicator data are determined by the target prediction model; the data processing device also includes a training unit 503, which is further used for:

[0239] Obtain sample data pairs; sample data pairs include sample visual data, sample text description information, and consistency labels;

[0240] The visual data of the samples is processed by feature extraction to obtain the visual feature vector of the samples, and the text description information of the samples is processed by feature extraction to obtain the text feature vector of the samples.

[0241] Based on the difference calculation rules, the sample difference indicator data between the sample visual feature vector and the sample text feature vector are determined, and based on the similarity calculation rules, the sample similarity indicator data between the sample visual feature vector and the sample text feature vector are determined.

[0242] The first difference data is determined based on the sample difference indicator data and the consistency label, and the second difference data is determined based on the sample similarity indicator data and the consistency label;

[0243] Based on the first and second difference data, the target difference data is determined, and the initial prediction model is tuned using the target difference data to obtain the target prediction model.

[0244] In one possible implementation, the training unit 503, when used to determine the target difference data based on the first difference data and the second difference data, is specifically used for:

[0245] The first weighted difference data is determined based on the first difference data and the first weight parameter, and the second weighted difference data is determined based on the second difference data and the second weight parameter; wherein, the first difference data is calculated using the cross-entropy loss function based on the sample difference index data and the consistency label, and the second difference data is calculated using the mean squared error loss function based on the sample similarity index data and the consistency label;

[0246] The target difference data is obtained by summing the first weighted difference data and the second weighted difference data.

[0247] In one possible implementation, the processing unit 502 is further configured to:

[0248] In response to the upload request from the second terminal device, the target data pair is parsed from the upload request; the target data pair includes visual data and text description information.

[0249] Based on the consistency detection results of visual data and text description information, determine the consistency label of the target data pair;

[0250] The target data pairs and their consistency tags are associated and stored in the information database;

[0251] The consistency label includes a first label for indicating data matching and a second label for indicating data mismatch; the consistency data label is used to determine the push priority based on the consistency label of the candidate response data during the process of pushing candidate response data to a third terminal device.

[0252] It should be noted that the functions of each unit of the data processing device in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments. The specific implementation process can be referred to the relevant descriptions in the method embodiments of this application, which will not be repeated here.

[0253] According to another embodiment of this application, Figure 5 The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0254] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 3 Computer programs for the steps involved in some or all of the methods shown, to construct, for example... Figure 5 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable storage medium, loaded onto the aforementioned computing device via the computer-readable storage medium, and executed therein.

[0255] Based on the above methods and apparatus embodiments, this application provides a computer device. Please refer to... Figure 6 This figure is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6The computer device shown includes at least a processor 601, an input interface 602, an output interface 603, and a computer-readable storage medium 604. The processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 can be connected via a bus or other means.

[0256] Computer-readable storage medium 604 can be stored in the memory of a computer device. Computer-readable storage medium 604 is used to store computer programs, including program instructions. Processor 601 is used to execute the computer program stored in computer-readable storage medium 604. Processor 601 (or CPU (Central Processing Unit)) is the computing and control core of the computer device; it is suitable for implementing computer programs, specifically for loading and executing computer programs to achieve the above-mentioned functions. Figure 2 or Figure 3 The method flow is shown.

[0257] This application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space for storing the operating system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by a processor. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0258] The computer equipment can be the above. Figure 1 The data processing system shown includes server 102. In a specific implementation, processor 601 can load and execute a computer program stored in computer-readable storage medium 604 to implement the corresponding steps of the data processing methods in the various method embodiments of this application.

[0259] In specific implementations, the processor 601, input interface 602, output interface 603, and computer-readable storage medium 604 described in the embodiments of this application can execute the embodiments of this application. Figure 2 or Figure 3 The implementation methods described in the relevant embodiments of the provided method can also be used to execute the embodiments of this application. Figure 5 The implementation methods described in the relevant embodiments of the provided device will not be repeated here.

[0260] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, systems, and computer devices can be implemented in other ways. The embodiments described above are merely illustrative, and the division of units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0261] This application also provides a computer program product, which includes program instructions stored in a computer-readable storage medium. A processor of a computer device reads the program instructions from the computer-readable storage medium and executes the program instructions, causing the computer device to perform the aforementioned data processing method, which will not be described in detail here.

[0262] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0263] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in or transmitted through a computer-readable storage medium. The program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0264] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0265] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0266] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Visual feature vectors are obtained by performing feature extraction on visual data, and text feature vectors are obtained by performing feature extraction on text description information. The difference indication data between the visual feature vector and the text feature vector is determined according to the difference calculation rules, and the similarity indication data between the visual feature vector and the text feature vector is determined according to the similarity calculation rules. Based on the difference indicator data and the similarity indicator data, a consistency detection result is determined between the visual data and the text description information; the consistency detection result is used to indicate the degree of matching between the visual data and the text description information.

2. The method as described in claim 1, characterized in that, The method further includes: In response to a search request from a first terminal device for search text, multiple candidate response data corresponding to the search text are determined from an information database; the multiple candidate response data includes target response data, and the target response data includes the visual data and the text description information; When the consistency detection result indicates that the visual data and the text description information match, the target response data is returned to the first terminal device so that the target response data is displayed on the first terminal device.

3. The method as described in claim 1, characterized in that, The step of determining the difference indication data between the visual feature vector and the text feature vector according to the difference calculation rule includes: The visual feature vector is processed by vector mapping according to the first parameter matrix to obtain the first mapping vector; The text feature vector is processed by vector mapping according to the second parameter matrix to obtain the second mapped vector; The first mapping vector and the second mapping vector are processed by vector difference calculation to obtain the difference vector; The difference indicator data between the visual feature vector and the text feature vector is determined based on the difference vector; wherein, the difference indicator data includes the classification prediction value obtained by calling a classifier to classify the difference vector.

4. The method as described in claim 3, characterized in that, The step of determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules includes: The first mapping vector and the second mapping vector are processed to calculate the similarity between the visual feature vector and the text feature vector, thereby obtaining similarity indication data between them.

5. The method as described in claim 3, characterized in that, The step of determining the similarity indicator data between the visual feature vector and the text feature vector according to the similarity calculation rules includes: The visual feature vector is processed by vector mapping according to the third parameter matrix to obtain the third mapping vector; The text feature vector is processed by vector mapping according to the fourth parameter matrix to obtain the fourth mapping vector; The third mapping vector and the fourth mapping vector are processed to calculate the similarity between the visual feature vector and the text feature vector, thereby obtaining similarity indication data between them.

6. The method according to any one of claims 1-5, characterized in that, The step of determining the consistency detection result between the visual data and the text description information based on the difference indicator data and the similarity indicator data includes: The difference indicator data and the similarity indicator data are fused to obtain fused indicator data; wherein the data fusion processing is performed using a mean algorithm or a network layer. Based on the fusion indication data, the consistency detection result of the visual data and the text description information is determined.

7. The method according to any one of claims 1-5, characterized in that, The difference indicator data and the similarity indicator data are determined by the target prediction model; the method further includes: Acquire sample data pairs; the sample data pairs include sample visual data, sample text description information, and consistency labels; The visual data of the sample is processed by feature extraction to obtain a visual feature vector of the sample, and the text description information of the sample is processed by feature extraction to obtain a text feature vector of the sample. The sample difference indicator data between the sample visual feature vector and the sample text feature vector is determined according to the difference calculation rule, and the sample similarity indicator data between the sample visual feature vector and the sample text feature vector is determined according to the similarity calculation rule. First difference data is determined based on the sample difference indicator data and the consistency label, and second difference data is determined based on the sample similarity indicator data and the consistency label; Based on the first difference data and the second difference data, target difference data is determined, and the initial prediction model is tuned using the target difference data to obtain the target prediction model.

8. The method as described in claim 7, characterized in that, The step of determining the target difference data based on the first difference data and the second difference data includes: A first weighted difference data is determined based on the first difference data and the first weight parameter, and a second weighted difference data is determined based on the second difference data and the second weight parameter; wherein, the first difference data is calculated using the cross-entropy loss function based on the sample difference index data and the consistency label, and the second difference data is calculated using the mean squared error loss function based on the sample similarity index data and the consistency label; The first weighted difference data and the second weighted difference data are summed to obtain the target difference data.

9. The method according to any one of claims 1-5, characterized in that, The method further includes: In response to an upload request from a second terminal device, a target data pair is parsed from the upload request; the target data pair includes the visual data and the text description information. Based on the consistency detection results of the visual data and the text description information, the consistency label of the target data pair is determined; The target data pair and its consistency tag are associated and stored in the information database; The consistency label includes a first label for indicating data matching and a second label for indicating data mismatch; the consistency data label is used to determine the push priority based on the consistency label of the candidate response data during the process of pushing candidate response data to a third terminal device.

10. A data processing apparatus, characterized in that, The device includes: The feature extraction unit is used to perform feature extraction processing on visual data to obtain visual feature vectors, and to perform feature extraction processing on text description information to obtain text feature vectors. The processing unit is configured to determine the difference indication data between the visual feature vector and the text feature vector according to the difference calculation rules, and to determine the similarity indication data between the visual feature vector and the text feature vector according to the similarity calculation rules; The processing unit is further configured to determine the consistency detection result between the visual data and the text description information based on the difference indicator data and the similarity indicator data; the consistency detection result is used to indicate the degree of matching between the visual data and the text description information.

11. A computer device, characterized in that, The computer device includes: A processor is a tool for implementing computer programs. A computer-readable storage medium storing a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1-9.

Images