Training method of correlation discrimination model and correlation discrimination method and device

Abstract

The invention provides a training method of a correlation discrimination model and a correlation discrimination method and device, and relates to the technical field of computers, in particular to the application fields of machine learning, large language models, multi-modal models, generative search, search engines, information retrieval, visual content search and the like. According to the specific implementation scheme, a search word sample and an image snapshot sample are obtained; wherein the image snapshot sample is an image snapshot of the visual content sample; processing the search word sample and the image snapshot sample by using a correlation discrimination model to obtain a correlation reasoning result of the search word sample and the image snapshot sample; and based on the correlation reasoning result and the correlation representation label of the search word sample and the image snapshot sample, training a correlation discrimination model to obtain a trained correlation discrimination model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to application fields such as machine learning, large language models, multimodal models, generative search, search engines, information retrieval, and visual content search, and specifically to a training method for a relevance discrimination model, a relevance discrimination method, and a device. Background Art

[0002] In the field of information retrieval (e.g., visual content search), the relevance evaluation result refers to the degree of relevance between the search term and the search result. Specifically, after the search system receives the search term sent by the user, it needs to evaluate the relevance between the search term and multiple candidate search results, and based on this, filter out at least one target search result from the multiple candidate search results, and then send the target search result to the user. Summary of the invention

[0003] The present disclosure provides a training method for a correlation discrimination model, a correlation discrimination method and a device.

[0004] According to a first aspect of the present disclosure, a method for training a correlation discrimination model is provided, comprising:

[0005] Obtaining a search term sample and an image snapshot sample; wherein the image snapshot sample is an image snapshot of the visual content sample;

[0006] Using the correlation discrimination model, the search word samples and the image snapshot samples are processed to obtain the correlation reasoning results between the search word samples and the image snapshot samples;

[0007] Based on the correlation reasoning results and the correlation representation labels between the search term samples and the image snapshot samples, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0008] According to a second aspect of the present disclosure, a correlation determination method is provided, comprising:

[0009] Obtaining a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of the target visual content;

[0010] The target search term and the target image snapshot are processed by using the trained relevance discrimination model to obtain the relevance evaluation result of the target search term and the target image snapshot.

[0011] According to a third aspect of the present disclosure, a training device for a correlation discrimination model is provided, comprising:

[0012] A first data acquisition unit is used to acquire a search word sample and an image snapshot sample; wherein the image snapshot sample is an image snapshot of a visual content sample;

[0013] A second data acquisition unit is used to process the search word sample and the image snapshot sample using a correlation discrimination model to obtain a correlation inference result between the search word sample and the image snapshot sample;

[0014] The model training unit is used to train the relevance discrimination model based on the relevance inference result and the relevance representation label of the search word sample and the image snapshot sample to obtain a trained relevance discrimination model.

[0015] According to a fourth aspect of the present disclosure, a correlation determination device is provided, comprising:

[0016] A data acquisition unit, used to acquire a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of a target visual content;

[0017] The relevance discrimination unit is used to process the target search term and the target image snapshot using the trained relevance discrimination model to obtain a relevance evaluation result between the target search term and the target image snapshot.

[0018] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0019] at least one processor;

[0020] a memory communicatively coupled to the at least one processor;

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided in the first aspect of the present disclosure.

[0022] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided according to the first aspect of the present disclosure.

[0023] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method provided according to the first aspect of the present disclosure when executed by a processor.

[0024] The present disclosure can improve the accuracy of relevance evaluation results in the field of visual content search.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 A flowchart of a method for training a correlation discrimination model provided in an embodiment of the present disclosure;

[0028] Figure 2 A flowchart for obtaining a correlation reasoning result provided in an embodiment of the present disclosure;

[0029] Figure 3 A flowchart of a correlation determination method provided in an embodiment of the present disclosure;

[0030] Figure 4 A schematic diagram of a target image snapshot provided by an embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of an application scenario of a training method for a correlation discrimination model provided in an embodiment of the present disclosure;

[0032] Figure 6 A schematic diagram of an application scenario of a correlation determination method provided by an embodiment of the present disclosure;

[0033] Figure 7 A schematic structural block diagram of a training device for a correlation discrimination model provided in an embodiment of the present disclosure;

[0034] Figure 8 A schematic structural block diagram of a correlation determination device provided in an embodiment of the present disclosure;

[0035] Fig. 9 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0037] As described in the background technology, in the field of information retrieval (e.g., visual content search), the relevance evaluation result refers to the degree of relevance between the search term and the search result. Specifically, after the search system receives the search term sent by the user, it needs to evaluate the relevance between the search term and multiple candidate search results, and based on this, screen out at least one target search result from the multiple candidate search results, and then send the target search result to the user.

[0038] At present, in the field of visual content search, one of the commonly used relevance judgment methods is: obtain the target search term and the target visual content, and extract the text content included in the target visual content, and then use a simple relevance algorithm to evaluate the relevance of the target search term and the text content to obtain the relevance evaluation result of the target search term and the target visual content.

[0039] However, the inventors have found that the existing correlation determination method has many shortcomings. On the one hand, it only uses a simple correlation algorithm to evaluate the correlation between the target search term and the text content, but cannot well explore the deep relationship between the target search term and the text content; on the other hand, the text content included in the target visual content is difficult to accurately and comprehensively reflect the constituent elements of the target visual content, which in turn limits the reliability of the correlation evaluation results. Ultimately, it affects the accuracy of the correlation evaluation results.

[0040] In view of the above problems, the embodiments of the present disclosure provide a training method for a correlation discrimination model, which can be applied to electronic devices. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a mainframe computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing unit or other similar computing devices. Figure 1 The flowchart diagram shown illustrates a training method for a correlation discrimination model provided by an embodiment of the present disclosure. It should be noted that although a logical order is shown in the flowchart diagram, in some cases, the steps shown or described in the flowchart can also be performed in other orders.

[0041] Step S101, obtaining search term samples and image snapshot samples.

[0042] The search term samples may be user search terms collected from the search system; the image snapshot samples may be image snapshots of the visual content samples, i.e., static views. Here, the visual content samples may be static visual content or dynamic visual content (e.g., animated images, videos, etc.) collected from the search system.

[0043] In a specific example, the visual content sample may be a web page advertisement, and correspondingly, the image snapshot sample may be a landing page snapshot of the web page advertisement. The web page advertisement may be a static advertisement or a dynamic advertisement (e.g., an animated image advertisement, a video advertisement, etc.), which is not limited in the embodiments of the present disclosure.

[0044] Step S102: using a correlation discrimination model, processing the search word sample and the image snapshot sample to obtain a correlation inference result between the search word sample and the image snapshot sample.

[0045] The relevance discrimination model may be a multimodal large language model (MLLM). Here, the MLLM is pre-trained and has general language knowledge, world knowledge, professional knowledge in various fields, etc. Based on this, after obtaining the search word sample and the image snapshot sample, the relevance discrimination model may be used to process the search word sample and the image snapshot sample to obtain the relevance inference result of the search word sample and the image snapshot sample. The relevance inference result may be used to characterize the degree of inferential relevance between the search word sample and the visual content sample.

[0046] Step S103 , based on the correlation reasoning result and the correlation representation labels of the search term samples and the image snapshot samples, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0047] The correlation characterization tag can be used to characterize the actual correlation between the search term sample and the visual content sample. In addition, in the embodiment of the present disclosure, the correlation characterization tag has the same data composition as the correlation reasoning result, and the correlation characterization tag can be obtained manually, which is not described in detail in the embodiment of the present disclosure.

[0048] In one example, after obtaining the correlation reasoning result and the correlation characterization label, a target loss value between the correlation reasoning result and the correlation characterization label can be obtained, and based on the target loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model. For example, based on the target loss value, the model parameters of the correlation discrimination model are adjusted to obtain a trained correlation discrimination model.

[0049] By adopting the training method of the correlation discrimination model provided by the embodiment of the present disclosure, it is possible to obtain search word samples and image snapshot samples (that is, image snapshots of visual content samples), and use the correlation discrimination model to process the search word samples and image snapshot samples to obtain the correlation reasoning results of the search word samples and the image snapshot samples, and then based on the correlation reasoning results and the correlation characterization labels of the search word samples and the image snapshot samples, the correlation discrimination model is trained to obtain a trained correlation discrimination model. Thereafter, the trained correlation discrimination model can be used for correlation discrimination. For example, the target search word and the target image snapshot (that is, the image snapshot of the target visual content) can be obtained, and the target search word and the target image snapshot can be processed using the trained correlation discrimination model to obtain the correlation evaluation result of the target search word and the target image snapshot. In this process, on the one hand, the trained correlation discrimination model can well mine the deep relationship between the target search word and the target image snapshot; on the other hand, the target image snapshot can accurately and comprehensively reflect the various components in the target visual content, and therefore, the accuracy of the correlation evaluation result can be improved.

[0050] As mentioned above, in the embodiments of the present disclosure, the visual content sample can be static visual content or dynamic visual content. Based on this, in the embodiments of the present disclosure, when the visual content sample is dynamic visual content, multiple preliminary snapshot images can be obtained based on the dynamic visual content, and image snapshot samples can be obtained based on the multiple preliminary snapshots.

[0051] In one example, multiple preliminary snapshot images can be captured from the dynamic visual content at a preset time interval, and the multiple preliminary snapshot images can be deduplicated to obtain multiple intermediate snapshot images, and then image snapshot samples can be obtained based on the multiple intermediate snapshot images. For example, the multiple intermediate snapshot images can be spliced ​​in the order of capture time to obtain the image snapshot sample.

[0052] Further, in some optional implementations, the step S102 of “processing the search word sample and the image snapshot sample to obtain the correlation inference result between the search word sample and the image snapshot sample” may include:

[0053] Obtain visual interpretation information for a sample of image snapshots;

[0054] Based on the search word sample and the visual interpretation information, a correlation inference value between the search word sample and the image snapshot sample and an inference result of the explanation information for the correlation inference value are obtained;

[0055] Based on the relevance inference value and the explanation information inference result, the relevance inference result between the search word sample and the image snapshot sample is obtained.

[0056] The visual interpretation information may include description information and / or understanding information.

[0057] Here, the description information can be used to describe the various components (e.g., text content, visual elements, etc.) in the image snapshot sample. Specifically, the description information can be used to directly describe the various components in the image snapshot sample, and / or to describe the structural features of the various components in the image snapshot sample (e.g., the position, size, color, format, etc. of each component). The understanding information is used to characterize the understanding results for the image snapshot sample. Specifically, taking the visual content sample as a web page advertisement as an example, the understanding results for the image snapshot sample (the landing page snapshot of the web page advertisement) may include product characteristics, brand positioning, target audience, etc.

[0058] Based on the above, in one example, “obtaining visual interpretation information for an image snapshot sample” may include:

[0059] Obtain description and understanding information for a sample of image snapshots;

[0060] Based on the description information and understanding information, visual interpretation information for the image snapshot samples is obtained.

[0061] In a specific example, the image snapshot sample can be preprocessed (e.g., scaled, sliced, cropped, etc.) to obtain a preprocessed image snapshot sample, and the preprocessed image snapshot sample can be encoded using an image encoder to obtain an image feature vector, and then based on the image feature vector, description information and understanding information for the image snapshot sample can be obtained, and based on the description information and understanding information, visual interpretation information for the image snapshot sample can be obtained. For example, the description information and understanding information are combined to obtain visual interpretation information for the image snapshot sample.

[0062] Based on the above, it can be understood that the visual interpretation information for the image snapshot sample may include description information and understanding information, and thus the comprehensiveness of the content of the visual interpretation information can be ensured.

[0063] After obtaining the visual interpretation information for the image snapshot sample, the correlation inference value between the search word sample and the image snapshot sample and the explanation information inference result for the correlation inference value can be obtained based on the search word sample and the visual interpretation information.

[0064] Among them, the correlation inference value can be a probability value located in the numerical interval [0, 1] (wherein, the larger the correlation inference value, the greater the correlation between the search term sample and the image snapshot sample, and correspondingly, the smaller the correlation inference value, the smaller the correlation between the search term sample and the image snapshot sample); the explanatory information reasoning result can be an explanation of the reason for generating the correlation inference value, so as to explain why the correlation between the search term sample and the image snapshot sample can be quantified by the correlation inference value.

[0065] In one example, “obtaining a correlation inference value between the search term sample and the image snapshot sample based on the search term sample and the visual interpretation information, and an inference result of the explanation information for the correlation inference value” may include:

[0066] Based on the search word samples and visual interpretation information, the information aggregation results are obtained;

[0067] Based on the information aggregation results, the correlation inference value between the search word sample and the image snapshot sample is obtained;

[0068] Based on the information aggregation result and the relevance inference value, an explanation information inference result for the relevance inference value is obtained.

[0069] In a specific example, the search term sample can be initially converted to obtain a plurality of discrete, processable first basic units (that is, the search term sample is tokenized), and the word embedding module is used to perform a secondary conversion on the plurality of first basic units to obtain a first text vector, and the visual interpretation information can be initially converted to obtain a plurality of discrete, processable second basic units (that is, the visual interpretation information is tokenized), and the word embedding module is used to perform a secondary conversion on the plurality of second basic units to obtain a second text vector. Thereafter, based on the first text vector and the second text vector, an information aggregation result is obtained. For example, the first text vector and the second text vector can be concatenated to obtain an information aggregation result.

[0070] After obtaining the information aggregation results, the relevance inference value between the search word sample and the image snapshot sample can be obtained based on the information aggregation results, and the explanation information inference result for the relevance inference value can be obtained based on the information aggregation results and the relevance inference value. In this process, no complicated calculation process is involved, so the training efficiency of the relevance discrimination model can be improved.

[0071] Through the above method, in the embodiment of the present disclosure, on the one hand, visual interpretation information for the image snapshot sample can be obtained, so as to use the visual interpretation information to more accurately and comprehensively reflect the various components in the target visual content; on the other hand, after obtaining the visual interpretation information for the image snapshot sample, the correlation inference value between the search word sample and the image snapshot sample and the explanation information inference result for the correlation inference value can be obtained based on the search word sample and the visual interpretation information, and the correlation inference result between the search word sample and the image snapshot sample can be obtained based on the correlation inference value and the explanation information inference result. In other words, the correlation inference result includes not only the correlation inference value, but also the explanation information inference result, so the interpretability of the correlation inference result can be improved.

[0072] The following, combined with Figure 2 , taking the correlation discrimination model as MLLM as an example, the above contents included in step S102 are further explained.

[0073] After obtaining the search word samples and the image snapshot samples, first, the search word samples and the image snapshot samples may be simultaneously input into the MLLM.

[0074] Thereafter, the MLLM may be used to obtain visual interpretation information for the image snapshot sample. Specifically, the MLLM may be used to obtain description information and understanding information for the image snapshot sample, and based on the description information and understanding information, the visual interpretation information for the image snapshot sample may be obtained.

[0075] Next, MLLM can be used to obtain information aggregation results based on search word samples and visual interpretation information, and based on the information aggregation results, a correlation inference value between the search word samples and the image snapshot samples can be obtained, and then based on the information aggregation results and the correlation inference value, an explanation information inference result for the correlation inference value can be obtained. In addition, it should be noted that in the embodiment of the present disclosure, since the correlation inference value can be a probability value located in the numerical interval [0, 1], therefore, in actual implementation, after MLLM obtains the information aggregation results based on the search word samples and the visual interpretation information, MLLM can call a preset fully connected layer to process the information aggregation results, obtain the correlation inference value between the search word samples and the image snapshot samples, and then MLLM can obtain the explanation information inference result for the correlation inference value based on the information aggregation results and the correlation inference value.

[0076] Finally, MLLM is used to obtain the relevance inference results of the search term samples and the image snapshot samples based on the relevance inference value and the explanation information inference results.

[0077] Further, in some optional implementations, “training the relevance discrimination model based on the relevance inference result and the relevance characterization label between the search term sample and the image snapshot sample to obtain a trained relevance discrimination model” may include:

[0078] Obtaining a first loss value based on a correlation inference value included in the correlation inference result and a correlation characterization value included in the correlation characterization tag;

[0079] Obtaining a second loss value based on the explanation information reasoning result included in the correlation reasoning result and the explanation information representation result included in the correlation representation tag;

[0080] Based on the first loss value and the second loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0081] In one example, a loss value between a correlation inference value included in the correlation inference result and a correlation characterization value included in the correlation characterization tag can be calculated by a first preset loss function as a first loss value. The process can be characterized as follows:

[0082] L cls (U) = -log(softmax(F cls W cls ))

[0083] Among them, L ccls (U) is used to characterize the first loss value; F cls Used to represent the correlation characterization value included in the correlation characterization tag; u (n+1) ~u m It is used to characterize the correlation inference value included in the correlation inference result; θ is used to characterize the model parameters of the correlation discrimination model.

[0084] In one example, a second preset loss function may be used to calculate a loss value between the explanation information inference result included in the correlation inference result and the explanation information representation result included in the correlation representation tag as the second loss value. This process may be characterized as follows:

[0085]

[0086] Among them, L gen (U) is used to characterize the second loss value; u1~u n u is used to represent the interpretation information representation result included in the correlation representation tag, specifically used to represent multiple fragments of the interpretation information representation result included in the correlation representation tag; (n+1) ~u mIt is used to characterize the explanation information reasoning result included in the correlation reasoning result, and specifically used to characterize multiple slices of the explanation information reasoning result included in the correlation reasoning result; θ is used to characterize the model parameters of the correlation discrimination model.

[0087] After obtaining the first loss value and the second loss value, the correlation discrimination model can be trained based on the first loss value and the second loss value to obtain a trained correlation discrimination model. In an example, the correlation discrimination model can be trained based on the first loss value and the second loss value to obtain a trained correlation discrimination model in the following manner:

[0088] Obtaining a first fusion weight corresponding to the first loss value and a second fusion weight corresponding to the second loss value;

[0089] Based on the first fusion weight and the second fusion weight, the first loss value and the second loss value are fused to obtain a target loss value;

[0090] Based on the target loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0091] The first fusion weight and the second fusion weight may be a preset weight value respectively, and the sum of the first fusion weight and the second fusion weight is 1.

[0092] In a specific example, the first fusion weight and the second fusion weight may be respectively a preset weight value set based on historical experience, or may be respectively a preset weight value set based on a training experiment, which is not limited in the embodiment of the present disclosure.

[0093] In another example, the first fusion weight and the second fusion weight can be determined based on the content type of the visual content sample. For example, when the visual content sample belongs to commercial visual content, a higher preset weight value can be assigned to the first fusion weight, and correspondingly, a lower preset weight value can be assigned to the second fusion weight (for example, the first fusion weight is 0.7 and the second fusion weight is 0.3); when the visual content sample belongs to entertainment visual content, the first fusion weight and the second fusion weight can be assigned the same preset weight value (that is, the first fusion weight is 0.5 and the second fusion weight is 0.5), or, a lower preset weight value can be assigned to the first fusion weight, and correspondingly, a higher preset weight value can be assigned to the second fusion weight (for example, the first fusion weight is 0.4 and the second fusion weight is 0.6).

[0094] In a specific example, “based on the first fusion weight and the second fusion weight, the first loss value and the second loss value are fused to obtain the target loss value” can be represented as:

[0095] L(U)=αL cls (U)+(1-α)L gen (U)

[0096] Among them, L(U) is used to represent; α is used to represent the first fusion weight; 1-α is used to represent the second fusion weight; L ccls (U) is used to characterize the first loss value; L gen (U) is used to characterize the second loss value.

[0097] Through the above manner, in the embodiment of the present disclosure, a first loss value can be obtained based on the correlation inference value included in the correlation inference result and the correlation characterization value included in the correlation characterization label, and a second loss value can be obtained based on the explanation information inference result included in the correlation inference result and the explanation information characterization result included in the correlation characterization label, and the correlation discrimination model is trained based on the first loss value and the second loss value to obtain a trained correlation discrimination model. On the one hand, the process does not involve a complex calculation process, so the training efficiency of the correlation discrimination model can be improved; on the other hand, in the process, the first loss value between the correlation inference value and the correlation characterization value, and the second loss value between the explanation information inference result and the explanation information characterization result are considered at the same time. By calculating these two loss values ​​and optimizing the correlation discrimination model, the probability of overfitting can be reduced to a certain extent, so that the correlation discrimination model can maintain stable performance when facing new input data (for example, the target search terms and target image snapshots mentioned later).

[0098] It should be noted that, in the embodiment of the present disclosure, step S101, step S102 and step S103 can be executed repeatedly until the correlation discrimination model meets the convergence condition to obtain the final trained correlation discrimination model. The convergence condition can be that the target loss value is less than or equal to the preset loss value. Here, the preset loss value can be set according to the actual application requirements, and the embodiment of the present disclosure does not limit this.

[0099] The embodiment of the present disclosure provides a correlation determination method, which can be applied to electronic devices. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a large computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 3 The flowchart diagram shown illustrates a correlation determination method provided by an embodiment of the present disclosure. It should be noted that although the logical order is shown in the flowchart diagram, in some cases, the steps shown or described in the flowchart can also be performed in other orders.

[0100] Step S301, obtaining a target search term and a target image snapshot.

[0101] The target search term may be a user search term received in real time; the target image snapshot may be an image snapshot of the target visual content, that is, a static view. Here, the target visual content may be any one of a plurality of candidate visual contents; the plurality of candidate visual contents may be static visual contents or dynamic visual contents (e.g., animated images, videos, etc.) that are determined to be relevant to the target search term from the search system through a preliminary judgment strategy.

[0102] In a specific example, the target visual content may be a web page advertisement, and correspondingly, the target image snapshot may be a landing page snapshot of the web page advertisement. The web page advertisement may be a static advertisement or a dynamic advertisement (e.g., an animated image advertisement, a video advertisement, etc.), which is not limited in the embodiments of the present disclosure.

[0103] Step S302: Process the target search term and the target image snapshot using the trained relevance discrimination model to obtain a relevance evaluation result between the target search term and the target image snapshot.

[0104] The trained correlation discrimination model may be a correlation discrimination model trained by the aforementioned correlation discrimination model training method.

[0105] After obtaining the target search term and the target image snapshot, the target search term and the target image snapshot can be processed using the trained correlation discrimination model to obtain a correlation evaluation result between the target search term and the target image snapshot. The correlation evaluation result can be used to characterize the actual correlation between the target search term and the target visual content.

[0106] By adopting the training method of the correlation discrimination model provided by the embodiment of the present disclosure, the target search term and the target image snapshot can be obtained, and the target search term and the target image snapshot can be processed using the trained correlation discrimination model to obtain the correlation evaluation result of the target search term and the target image snapshot. In this process, on the one hand, the trained correlation discrimination model can well explore the deep relationship between the target search term and the target image snapshot; on the other hand, the target image snapshot can accurately and comprehensively reflect the various components in the target visual content, thereby improving the accuracy of the correlation evaluation result.

[0107] In addition, it should be noted that in the embodiment of the present disclosure, after receiving the user search term and determining, through a preliminary judgment strategy, multiple candidate visual content that is relevant to the target search term from the search system, each of the multiple candidate visual content can be used as the target visual content, and through the relevance judgment method described in steps S301 and S302, a relevance judgment result corresponding to the target visual content is obtained, and then multiple relevance judgment results corresponding one-to-one to the multiple candidate visual contents are obtained.

[0108] Finally, based on multiple relevance judgment results, at least one content to be sent is selected from multiple candidate visual contents and returned to the target user who sent the user search term.

[0109] Further, in some optional implementations, the step S302 of "processing the target search term and the target image snapshot to obtain a correlation evaluation result between the target search term and the target image snapshot" may include:

[0110] Obtaining actual interpretation information for the target image snapshot;

[0111] Based on the target search term and the actual interpretation information, an actual correlation representation value between the target search term and the target image snapshot and actual interpretation information for the actual correlation representation value are obtained;

[0112] Based on the actual relevance representation value and the actual interpretation information, the relevance evaluation result between the target search term and the target image snapshot is obtained.

[0113] In one example, “obtaining actual interpretation information for a target image snapshot” may include:

[0114] Obtaining actual description information and actual understanding information for the target image snapshot;

[0115] Based on the actual description information and the actual understanding information, actual interpretation information for the target image snapshot is obtained.

[0116] In one example, “obtaining, based on the target search term and the actual interpretation information, an actual correlation representation value between the target search term and the target image snapshot, and actual interpretation information for the actual correlation representation value” may include:

[0117] Based on the target search terms and the actual interpretation information, the actual information aggregation results are obtained;

[0118] Based on the actual information aggregation results, the actual correlation representation value between the target search term and the target image snapshot is obtained;

[0119] Based on the actual information aggregation result and the actual relevance characterization value, actual interpretation information for the actual relevance characterization value is obtained.

[0120] For the specific functions and examples of each step in the embodiments of the present disclosure, please refer to the relevant descriptions of the corresponding steps in the aforementioned training method embodiment of the correlation discrimination model, which will not be repeated here.

[0121] The following will illustrate the application process of a trained correlation discrimination model provided by an embodiment of the present disclosure through a specific example.

[0122] Assume that there is a target search term "car paint protection film, with self-repairing ability" and the target visual content is a web page advertisement of car paint protection film, and the corresponding target image snapshot is as follows: Figure 4 shown.

[0123] After the target search term and the target image snapshot are input into the trained relevance discrimination model, the target search term and the target image snapshot are processed using the trained relevance discrimination model, and the obtained relevance evaluation results of the target search term and the target image snapshot may include:

[0124] Actual correlation representation value: 0.9;

[0125] Actual explanatory information for the actual relevance characterization value: The target visual content shows the car paint protection film and its core features. Moreover, its text clearly describes that this car paint protection film has self-repairing ability - using nanotechnology, it can automatically repair minor scratches in a warm environment and restore the paint surface to a perfect state. These descriptions are completely consistent with the target search terms "car paint protection film" and "self-repairing ability". Therefore, there is a high correlation between the target visual content and the target search terms.

[0126] See also Figure 5 , which is a schematic diagram of an application scenario of a training method for a correlation discrimination model provided in an embodiment of the present disclosure.

[0127] The training method of the correlation discrimination model provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device may be a server or a terminal device. Here, the terminal device may be a workbench, a mainframe computer, a conventional computer (e.g., a desktop computer, a laptop computer, a vehicle-mounted computer, etc.), a personal digital processing or other similar computing device. Here, the electronic device is used to:

[0128] Obtaining a search term sample and an image snapshot sample; wherein the image snapshot sample is an image snapshot of the visual content sample;

[0129] Using the correlation discrimination model, the search word samples and the image snapshot samples are processed to obtain the correlation reasoning results between the search word samples and the image snapshot samples;

[0130] Based on the correlation reasoning results and the correlation representation labels between the search term samples and the image snapshot samples, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0131] It should be noted that in the embodiments of the present disclosure, Figure 5 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 5 Various obvious changes and / or substitutions are made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0132] See also Figure 6 , which is a schematic diagram of an application scenario of a correlation determination method provided in an embodiment of the present disclosure.

[0133] The correlation determination method provided by the embodiment of the present disclosure is applied to an electronic device. The electronic device may be a server or a terminal device. The terminal device may be a workbench, a mainframe computer, a conventional computer (e.g., a desktop computer, a laptop computer, a vehicle-mounted computer, etc.), a personal digital processing device or other similar computing device. The electronic device is used to:

[0134] Obtaining a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of the target visual content;

[0135] The target search term and the target image snapshot are processed by using the trained relevance discrimination model to obtain the relevance evaluation result of the target search term and the target image snapshot.

[0136] It should be noted that in the embodiments of the present disclosure, Figure 6 The scene diagram shown is only illustrative and not restrictive. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions are made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0137] In order to better implement the training method of the aforementioned correlation discrimination model, the embodiment of the present disclosure also provides a training device 700 for the correlation discrimination model, which can be integrated into an electronic device. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a mainframe computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 7 The schematic structural block diagram shown illustrates a training device 700 for a correlation discrimination model provided by the disclosed embodiment.

[0138] The training device 700 of the correlation discrimination model includes:

[0139] The first data acquisition unit 701 is used to acquire search word samples and image snapshot samples; wherein the image snapshot samples are image snapshots of visual content samples;

[0140] The second data acquisition unit 702 is used to process the search word sample and the image snapshot sample using a correlation discrimination model to obtain a correlation inference result between the search word sample and the image snapshot sample;

[0141] The model training unit 703 is used to train the relevance discrimination model based on the relevance inference result and the relevance representation label of the search word sample and the image snapshot sample to obtain a trained relevance discrimination model.

[0142] In some optional implementations, the second data acquisition unit 702 is used to:

[0143] Obtain visual interpretation information for a sample of image snapshots;

[0144] Based on the search word sample and the visual interpretation information, a correlation inference value between the search word sample and the image snapshot sample and an inference result of the explanation information for the correlation inference value are obtained;

[0145] Based on the relevance inference value and the explanation information inference result, the relevance inference result between the search word sample and the image snapshot sample is obtained.

[0146] In some optional implementations, the second data acquisition unit 702 is used to:

[0147] Obtain description and understanding information for a sample of image snapshots;

[0148] Based on the description information and understanding information, visual interpretation information for the image snapshot samples is obtained.

[0149] In some optional implementations, the second data acquisition unit 702 is used to:

[0150] Based on the search word samples and visual interpretation information, the information aggregation results are obtained;

[0151] Based on the information aggregation results, the correlation inference value between the search word sample and the image snapshot sample is obtained;

[0152] Based on the information aggregation result and the relevance inference value, an explanation information inference result for the relevance inference value is obtained.

[0153] In some optional implementations, the model training unit 703 is used to:

[0154] Obtaining a first loss value based on a correlation inference value included in the correlation inference result and a correlation characterization value included in the correlation characterization tag;

[0155] Obtaining a second loss value based on the explanation information reasoning result included in the correlation reasoning result and the explanation information representation result included in the correlation representation tag;

[0156] Based on the first loss value and the second loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

[0157] In some optional implementations, the model training unit 703 is used to:

[0158] Obtaining a first fusion weight corresponding to the first loss value and a second fusion weight corresponding to the second loss value;

[0159] Based on the first fusion weight and the second fusion weight, the first loss value and the second loss value are fused to obtain a target loss value;

[0160] Based on the target loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model. In the disclosed embodiment, the specific functions and examples of each unit in the training device 700 of the correlation discrimination model can be found in the relevant description of the corresponding steps in the aforementioned training method embodiment of the correlation discrimination model, which will not be repeated here.

[0161] In order to better implement the aforementioned correlation determination method, the embodiment of the present disclosure further provides a correlation determination device 800, which can be integrated into an electronic device. The electronic device can be a server or a terminal device. Here, the terminal device can be a workbench, a large computer, a conventional computer (for example, a desktop computer, a laptop computer, a car computer, etc.), a personal digital processing or other similar computing device. Figure 8 The schematic structural block diagram shown illustrates a correlation determination device 800 provided in the disclosed embodiment.

[0162] The correlation determination device 800 comprises:

[0163] The data acquisition unit 801 is used to acquire a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of a target visual content;

[0164] The relevance determination unit 802 is used to process the target search term and the target image snapshot using the trained relevance determination model to obtain a relevance evaluation result between the target search term and the target image snapshot.

[0165] In some optional implementations, the correlation determination unit 802 is used to:

[0166] Obtaining actual interpretation information for the target image snapshot;

[0167] Based on the target search term and the actual interpretation information, an actual correlation representation value between the target search term and the target image snapshot and actual interpretation information for the actual correlation representation value are obtained;

[0168] Based on the actual relevance representation value and the actual interpretation information, the relevance evaluation result between the target search term and the target image snapshot is obtained.

[0169] In some optional implementations, the correlation determination unit 802 is used to:

[0170] Obtaining actual description information and actual understanding information for the target image snapshot;

[0171] Based on the actual description information and the actual understanding information, actual interpretation information for the target image snapshot is obtained.

[0172] In some optional implementations, the correlation determination unit 802 is used to:

[0173] Based on the target search terms and the actual interpretation information, the actual information aggregation results are obtained;

[0174] Based on the actual information aggregation results, the actual correlation representation value between the target search term and the target image snapshot is obtained;

[0175] Based on the actual information aggregation result and the actual relevance characterization value, actual interpretation information for the actual relevance characterization value is obtained.

[0176] In the embodiment of the present disclosure, the specific functions and examples of each unit in the correlation determination device 800 can be found in the relevant descriptions of the corresponding steps in the aforementioned correlation determination method embodiment, and will not be repeated here.

[0177] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0178] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0179] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as vehicle-mounted computing devices, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0180] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0181] A number of components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of renderers, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the correlation discrimination model and / or the correlation discrimination method. For example, in some embodiments, the training method of the correlation discrimination model and / or the correlation discrimination method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the training method of the correlation discrimination model and / or the correlation discrimination method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured as the training method of the correlation discrimination model and / or the correlation discrimination method in any other appropriate manner (e.g., by means of firmware).

[0183] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data optimization device so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed completely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine or completely on a remote machine or server.

[0185] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a rendering device (e.g., a cathode ray tube (CRT) renderer or a liquid crystal display (LCD)) for rendering information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices are also used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0187] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), and the Internet.

[0188] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0189] The embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a training method for a correlation discrimination model and / or a correlation discrimination method.

[0190] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements a training method for a correlation discrimination model and / or a correlation discrimination method.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and this document does not limit this. In addition, in the present disclosure, relational terms such as "first", "second", "third", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. In addition, "multiple" in the present disclosure can be understood as at least two.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a correlation discrimination model, comprising: Acquire a search term sample and an image snapshot sample; wherein the image snapshot sample is an image snapshot of a visual content sample; Using a correlation discrimination model, the search word sample and the image snapshot sample are processed to obtain a correlation inference result between the search word sample and the image snapshot sample; Based on the correlation inference result and the correlation representation label between the search term sample and the image snapshot sample, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

2. The method according to claim 1, wherein: The processing of the search word sample and the image snapshot sample to obtain a correlation inference result between the search word sample and the image snapshot sample includes: Obtaining visual interpretation information for the image snapshot sample; Based on the search word sample and the visual interpretation information, obtaining a correlation inference value between the search word sample and the image snapshot sample, and an inference result of the explanation information for the correlation inference value; Based on the relevance inference value and the explanation information inference result, a relevance inference result between the search term sample and the image snapshot sample is obtained.

3. The method according to claim 2, wherein: The obtaining of visual interpretation information for the image snapshot sample includes: Obtaining description information and understanding information for the image snapshot sample; Based on the description information and the understanding information, visual interpretation information for the image snapshot sample is obtained.

4. The method according to claim 2, wherein: The obtaining, based on the search word sample and the visual interpretation information, a correlation inference value between the search word sample and the image snapshot sample, and an inference result of the explanation information for the correlation inference value, includes: Based on the search word sample and the visual interpretation information, obtaining an information aggregation result; Based on the information aggregation result, obtaining a correlation inference value between the search word sample and the image snapshot sample; Based on the information aggregation result and the relevance inference value, an explanation information inference result for the relevance inference value is obtained.

5. The method according to any one of claims 1 to 4, wherein: The training of the correlation discrimination model based on the correlation inference result and the correlation representation label between the search term sample and the image snapshot sample to obtain a trained correlation discrimination model includes: Obtaining a first loss value based on the correlation inference value included in the correlation inference result and the correlation characterization value included in the correlation characterization tag; Obtaining a second loss value based on the explanation information reasoning result included in the correlation reasoning result and the explanation information representation result included in the correlation representation tag; Based on the first loss value and the second loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

6. The method according to claim 5, wherein: The step of training the correlation discrimination model based on the first loss value and the second loss value to obtain a trained correlation discrimination model includes: Obtaining a first fusion weight corresponding to the first loss value and a second fusion weight corresponding to the second loss value; Based on the first fusion weight and the second fusion weight, the first loss value and the second loss value are fused to obtain a target loss value; Based on the target loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

7. A correlation determination method, comprising: Obtaining a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of the target visual content; The target search term and the target image snapshot are processed using a trained relevance discrimination model to obtain a relevance evaluation result between the target search term and the target image snapshot.

8. A training device for a correlation discrimination model, comprising: A first data acquisition unit is used to acquire a search word sample and an image snapshot sample; wherein the image snapshot sample is an image snapshot of a visual content sample; A second data acquisition unit is used to process the search word sample and the image snapshot sample by using a correlation discrimination model to obtain a correlation inference result between the search word sample and the image snapshot sample; The model training unit is used to train the relevance discrimination model based on the relevance inference result and the relevance characterization label between the search term sample and the image snapshot sample to obtain a trained relevance discrimination model.

9. The device according to claim 8, wherein: The second data acquisition unit is used for: Obtaining visual interpretation information for the image snapshot sample; Based on the search word sample and the visual interpretation information, obtaining a correlation inference value between the search word sample and the image snapshot sample, and an inference result of the explanation information for the correlation inference value; Based on the relevance inference value and the explanation information inference result, a relevance inference result between the search term sample and the image snapshot sample is obtained.

10. The device according to claim 9, wherein: The second data acquisition unit is used for: Obtaining description information and understanding information for the image snapshot sample; Based on the description information and the understanding information, visual interpretation information for the image snapshot sample is obtained.

11. The device according to claim 9, wherein: The second data acquisition unit is used for: Based on the search word sample and the visual interpretation information, obtaining an information aggregation result; Based on the information aggregation result, obtaining a correlation inference value between the search word sample and the image snapshot sample; Based on the information aggregation result and the relevance inference value, an explanation information inference result for the relevance inference value is obtained.

12. The device according to any one of claims 8 to 11, wherein: The model training unit is used for: Obtaining a first loss value based on the correlation inference value included in the correlation inference result and the correlation characterization value included in the correlation characterization tag; Obtaining a second loss value based on the explanation information reasoning result included in the correlation reasoning result and the explanation information representation result included in the correlation representation tag; Based on the first loss value and the second loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

13. The device according to claim 12, wherein: The model training unit is used for: Obtaining a first fusion weight corresponding to the first loss value and a second fusion weight corresponding to the second loss value; Based on the first fusion weight and the second fusion weight, the first loss value and the second loss value are fused to obtain a target loss value; Based on the target loss value, the correlation discrimination model is trained to obtain a trained correlation discrimination model.

14. A correlation determination device, comprising: A data acquisition unit, used to acquire a target search term and a target image snapshot; wherein the target image snapshot is an image snapshot of a target visual content; The relevance determination unit is used to process the target search term and the target image snapshot using a trained relevance determination model to obtain a relevance evaluation result between the target search term and the target image snapshot.

15. An electronic device, comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images