Media information processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a media information processing method, device, electronic device and storage medium, which belongs to the field of network technology. In the embodiment of the present disclosure, the features of the first media information and the features of the second media information are respectively extracted based on the retrieval model; the first media information is information in a video mode or a text mode, and the second media information is different from the first media information in a different mode. Based on the features of the first media information and the features of the second media information, the first similarity information of the first media information relative to the second media information is determined. According to the first similarity information, the first probability information that the second media information matches the first media information is determined. The first similarity information is corrected based on the first probability information, and the target processing is performed based on the corrected first similarity information. In this way, the accuracy of subsequent target processing can be improved, thereby improving the accuracy of cross-modal retrieval and ensuring the retrieval effect of cross-modal retrieval.

Description

Technical Field

[0001] The present disclosure belongs to the field of network technology, and in particular relates to a media information processing method, device, electronic device and storage medium. Background Art

[0002] At present, cross-modal retrieval is often required, using media information in modality A to retrieve media information in modality B, or using media information in modality B to retrieve media information in modality A. For example, in a video search scenario, sometimes it is necessary to retrieve matching videos using text information in text modality. Alternatively, based on the video, text information matching the video is retrieved from a text library to add a text description to the video.

[0003] In the related art, in order to realize cross-modal media information retrieval, the similarity information between different modal media information is often calculated, and then subsequent processing is directly based on the similarity information. Due to the heterogeneity of different modal media information, this method sometimes leads to low accuracy of subsequent processing, resulting in poor retrieval effect. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a media information processing method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a media information processing method is provided, the method comprising:

[0006] Based on the retrieval model, the features of the first media information and the features of the second media information are respectively extracted; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information;

[0007] Determine first similarity matrix information of the first media information relative to the second media information based on the characteristics of the first media information media information and the characteristics of the second media information media information;

[0008] Determine first probability matrix information of matching the second media information with the first media information based on the first similarity matrix information;

[0009] The first similarity information matrix is ​​modified based on the first probability information matrix, and target processing is performed based on the modified first similarity information matrix; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model.

[0010] Optionally, the first similarity information is a first similarity matrix, and the first probability information is a first probability matrix; a row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row relative to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column relative to other first media information; determining, based on the first similarity information, the first probability information that the second media information matches the first media information includes:

[0011] For any similarity in the first similarity matrix, a probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs;

[0012] The first probability matrix is ​​generated according to the probabilities.

[0013] Optionally, calculating the probability corresponding to the similarity according to the similarity in the column to which the similarity belongs includes:

[0014] The similarity is normalized according to the similarities in the column to which the similarity belongs, so as to generate a probability corresponding to the similarity.

[0015] Optionally, the modifying the first similarity information based on the first probability information includes:

[0016] A point multiplication is performed based on the first similarity information and the first probability information to perform correction.

[0017] Optionally, when the retrieval model has been trained, performing target processing based on the corrected first similarity information includes:

[0018] Normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs;

[0019] An output value of the retrieval model is determined according to the target probability information.

[0020] Optionally, when the retrieval model has not completed training, the performing target processing based on the corrected first similarity information includes:

[0021] Calculating a first loss value according to the corrected first similarity information and the first probability information;

[0022] Based on the first loss value, a loss value of the retrieval model is determined.

[0023] Optionally, the method further includes:

[0024] Determining second probability information that the first media information matches the second media information according to second similarity information between the second media information and the first media information;

[0025] modifying the second similarity information based on the second probability information, and calculating a second loss value based on the modified second similarity information and the second probability information;

[0026] Determining the loss value of the retrieval model based on the first loss value includes: calculating the loss value of the retrieval model based on the first loss value and the second loss value.

[0027] According to a second aspect of the present disclosure, a media information processing device is provided, the device comprising:

[0028] an extraction module configured to extract features of first media information and features of second media information respectively based on a retrieval model; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information;

[0029] A first determining module is configured to determine first similarity information of the first media information relative to the second media information based on a feature of the first media information and a feature of the second media information;

[0030] a second determination module, configured to determine first probability information that the second media information matches the first media information according to the first similarity information;

[0031] An execution module is configured to modify the first similarity information based on the first probability information, and perform target processing based on the modified first similarity information; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model.

[0032] Optionally, the first similarity information is a first similarity matrix, and the first probability information is a first probability matrix; a row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row relative to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column relative to other first media information; the second determination module is specifically configured as follows:

[0033] For any similarity in the first similarity matrix, a probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs;

[0034] The first probability matrix is ​​generated according to the probabilities.

[0035] Optionally, the second determining module is further configured to:

[0036] The similarity is normalized according to the similarities in the column to which the similarity belongs, so as to generate a probability corresponding to the similarity.

[0037] Optionally, the execution module is specifically configured as follows:

[0038] A dot product is performed based on the first similarity matrix and the first probability matrix to perform matrix correction.

[0039] Optionally, when the retrieval model has been trained, the execution module is further configured to:

[0040] Normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs;

[0041] An output value of the retrieval model is determined according to the target probability information.

[0042] Optionally, when the retrieval model has not completed training, the execution module is further configured to:

[0043] Calculating a first loss value according to the corrected first similarity information and the first probability information;

[0044] Based on the first loss value, a loss value of the retrieval model is determined.

[0045] Optionally, the device further comprises:

[0046] a third determination module, configured to determine second probability information that the first media information matches the second media information based on second similarity information between the second media information and the first media information;

[0047] a calculation module, configured to modify the second similarity information based on the second probability information, and calculate a second loss value based on the modified second similarity information and the second probability information;

[0048] The execution module is further specifically configured to calculate the loss value of the retrieval model based on the first loss value and the second loss value.

[0049] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0050] processor;

[0051] a memory for storing instructions executable by the processor;

[0052] The processor is configured to execute the instructions to implement the method as described in any one of the first aspects.

[0053] According to a fourth aspect of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device executes the method as described in any one of the first aspects.

[0054] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product comprising readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, the electronic device executes the method as described in any one of the first aspects.

[0055] Compared with the related art, the present invention has the following advantages and positive effects:

[0056] The media information processing method provided by the embodiment of the present disclosure extracts the features of the first media information and the features of the second media information respectively based on the retrieval model; the first media information is information in video mode or text mode, and the second media information is different from the first media information in mode. Based on the features of the first media information and the features of the second media information, the first similarity information of the first media information relative to the second media information is determined. According to the first similarity information, the first probability information of the match between the second media information and the first media information is determined. The first similarity information is corrected based on the first probability information, and the target processing is performed based on the corrected first similarity information; the target processing includes generating the output of the retrieval model and / or determining the loss value of the retrieval model. In this way, the first probability information of the degree of match between the two is measured from the perspective of retrieving the first media information from the second media information, and the first similarity information is further corrected first, which can make the corrected first similarity information more accurate to a certain extent, and then improve the accuracy of subsequent target processing, thereby improving the accuracy of cross-modal retrieval and ensuring the retrieval effect of cross-modal retrieval.

[0057] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present disclosure. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0059] Figure 1 is a flow chart of steps of a media information processing method provided by an embodiment of the present disclosure;

[0060] Figure 2 is a search schematic diagram provided by the present disclosure;

[0061] Figure 3 is a schematic diagram of a processing process provided by an embodiment of the present disclosure;

[0062] Figure 4 is another processing process schematic diagram provided by an embodiment of the present disclosure;

[0063] Figure 5 is another processing process schematic diagram provided by an embodiment of the present disclosure;

[0064] Figure 6 is a block diagram of a media information processing device provided by an embodiment of the present disclosure;

[0065] Figure 7 is a block diagram of a device for media information processing according to an exemplary embodiment;

[0066] Figure 8 The invention is a block diagram of a device for processing media information according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] Figure 1 is a flow chart of the steps of a media information processing method provided by an embodiment of the present disclosure, such as Figure 1As shown, the method may include:

[0069] Step 101: extract features of first media information and features of second media information based on a retrieval model; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information.

[0070] Step 102: Determine first similarity information between the first media information and the second media information based on the characteristics of the first media information and the characteristics of the second media information.

[0071] In the embodiments of the present disclosure, the retrieval model can be a trained model or an untrained model. The retrieval model can be used for cross-modal retrieval. For example, the retrieval model can be used to retrieve second media information that matches the first media information from the second media information. Wherein, in the case where the first media information is a video in a video mode, the second media information can be a text in a text mode. In the case where the first media information is a text in a text mode, the second media information can be a video in a video mode. In this way, cross-modal retrieval between videos and texts can be achieved. For example, in one application scenario, retrieving videos with text can enable users to search for videos that meet their needs based on a video platform. In another application scenario, retrieving text with a video can easily determine the text that matches the video, so that matching text can be added to the video as a title, text description, and so on. Of course, the first media information and the second media information can also be media information of other modes, and the present disclosure does not limit this.

[0072] Furthermore, the number of first media information and the number of second media information may both be not less than 2. The first similarity information may be information in a rectangular form, and the elements in the first similarity information may represent the similarity of the first media information to the second media information when the matching second media information is retrieved based on the first media information. For example, each first media information and each second media information may be firstly subjected to Embedding processing to obtain the features of each first media information and the features of each second media information, and the features of the media information may be extracted based on the content described by the media information. Then, based on the features of each first media information and the features of each second media information, the similarity between each first media information and each second media information is calculated to form a first similarity matrix, thereby obtaining the first similarity information. Taking the first media information as a video and the second media information as a text as an example, the similarity may characterize the degree of matching between the content described by the video and the content described by the text.

[0073] Due to the heterogeneity of the content of the visual modality and the textual modality, that is, the content of the video description is usually very specific, while the description of the textual modality is usually abstract and vague. Therefore, directly calculating the similarity alone may lead to inaccurate retrieval, which will cause a bottleneck in the video text retrieval task. For example, a represents a video of a woman decorating her nails, b represents a video of a woman mixing nail polish and adding something to the nail polish, and c represents a video of a girl painting Easter patterns on her nails. Figure 2 is a search schematic diagram provided by the present disclosure, such as Figure 2 As shown in the figure, due to the heterogeneity of the two modal contents of video and text, text usually cannot describe every detail of the video. In this example, since the first sentence "A woman is decorating her finger nail" describes a much broader content than the other two sentences, it is very easy to match all videos of decorating nails, so the similarity between each video and the first text in the similarity matrix is ​​the greatest. But in fact, the first video should match the first sentence, the second video should match the second sentence, and the third video should match the third sentence. In other words, the highest similarity score should be in the diagonal part of the matrix, not in the same column.

[0074] Step 103: Determine first probability information that the second media information matches the first media information according to the first similarity information.

[0075] In the disclosed embodiment, the first probability information may be information in a matrix form, and the elements in the first probability information may represent the probability that the second media information matches the first media information. That is, the first probability information may measure the degree of match between the retrieved second media information and the first media information from the perspective of the second media information retrieving the first media information.

[0076] Step 104: modify the first similarity information based on the first probability information, and perform target processing based on the modified first similarity information; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model.

[0077] In the disclosed embodiment, since the first probability information can measure the degree of match between the two from the perspective of retrieving the first media information from the second media information, the first similarity information can be corrected based on the first probability information, which can make the corrected first similarity information more accurate to a certain extent, thereby ensuring the subsequent processing effect to a certain extent. Further, the output of the retrieval model can be information that can indicate the second media information that matches the first media information, for example, the output of the retrieval model can be the name, number, etc. of the second media information that matches the first media information. In this way, by determining the output of the retrieval model, cross-modal retrieval can be achieved. At the same time, by determining the output of the retrieval model based on the corrected first similarity information, the accuracy of the retrieval can be improved to a certain extent, thereby improving the retrieval effect. Alternatively, when the retrieval model is still in the training phase, the prediction ability of the retrieval model can be determined based on the corrected first similarity information, that is, the loss value of the retrieval model can be determined, thereby more accurately optimizing the retrieval model and improving the processing ability when the retrieval model is applied. In this way, when the retrieval model is used for retrieval, the accuracy of the model retrieval can be improved to a certain extent, thereby improving the retrieval effect.

[0078] In summary, the media information processing method provided by the embodiment of the present disclosure extracts the features of the first media information and the features of the second media information respectively based on the retrieval model; the first media information is information in video mode or text mode, and the second media information is different from the first media information in mode. Based on the features of the first media information and the features of the second media information, the first similarity information of the first media information relative to the second media information is determined. According to the first similarity information, the first probability information of the second media information matching the first media information is determined. The first similarity information is corrected based on the first probability information, and the target processing is performed based on the corrected first similarity information; the target processing includes generating the output of the retrieval model and / or determining the loss value of the retrieval model. In this way, the first probability information of the degree of matching between the two is measured from the perspective of retrieving the first media information from the second media information, and the first similarity information is further corrected first, which can make the corrected first similarity information more accurate to a certain extent, and then improve the accuracy of subsequent target processing, thereby improving the accuracy of cross-modal retrieval and ensuring the retrieval effect of cross-modal retrieval.

[0079] Optionally, the first similarity information may be a first similarity matrix, and the first probability information may be a first probability matrix. A row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row relative to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column relative to other first media information. That is, the similarity of the i-th row and j-th column may represent the similarity of the i-th first media information relative to the j-th second media information. Accordingly, the above-mentioned operation of determining the first probability information that the second media information matches the first media information based on the first similarity information may specifically include:

[0080] Step S21: for any similarity in the first similarity matrix, calculate the probability corresponding to the similarity according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs.

[0081] In the disclosed embodiment, the similarity in a column can represent the similarity of the second media information corresponding to the column to each first media information to a certain extent, and the greater the similarity, the higher the probability of matching. Therefore, the probability of the corresponding first media information matching the second media information can be determined vertically based on the similarity in units of columns, thereby obtaining the probability of the second media information corresponding to the column matching each first media information. The calculated probability can represent the prior probability.

[0082] Step S22: Generate the first probability matrix according to the probability.

[0083] In the embodiment of the present disclosure, the probability corresponding to the similarity may be used as the element of the row and column where the similarity is located, thereby forming a first probability matrix.

[0084] In the disclosed embodiment, after obtaining the first similarity matrix, for any similarity in the first similarity matrix, the probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs, and the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs. A first probability matrix is ​​generated according to the probability. In this way, the prior probability is pre-calculated in the vertical direction from the perspective of retrieving the first media information from the second media information in reverse order to generate the first probability matrix, which can ensure the subsequent correction effect to a certain extent.

[0085] Optionally, the above-mentioned operation of calculating the probability corresponding to the similarity according to the similarity in the column to which the similarity belongs may specifically include: step S31, normalizing the similarity according to the similarity in the column to which the similarity belongs to generate the probability corresponding to the similarity. The sum of the probabilities corresponding to the similarities in the same column is equal to 1. Specifically, the similarity and the similarity in the column to which the similarity belongs can be used as inputs of a preset softmax function to generate the probability corresponding to the similarity. Among them, the preset softmax function can map each similarity in the column to between zero and positive infinity, and then calculate the ratio between the mapping value corresponding to the similarity and the sum of the mapping values ​​corresponding to each similarity in the column to which it belongs, thereby achieving normalization and obtaining the probability corresponding to the similarity. In the embodiment of the present disclosure, the operation of generating the probability corresponding to the similarity can be achieved by only performing normalization processing based on the preset softmax function, thereby ensuring processing efficiency to a certain extent.

[0086] Optionally, the above-mentioned operation of correcting the first similarity information based on the first probability information may specifically include: step S41, performing a point multiplication based on the first similarity information and the first probability information for correction. In this step, any similarity in the first similarity matrix can be multiplied by the probability that the row and column in the first probability matrix are the same as the similarity, so as to achieve correction of the similarity. For example, the similarity of the i-th row and j-th column can be multiplied by the probability of the i-th row and j-th column. In the disclosed embodiment, correction can be achieved by performing a point multiplication based on the first similarity information and the first probability information to obtain the corrected first similarity information, thereby ensuring the correction efficiency to a certain extent.

[0087] Optionally, when the retrieval model has been trained, performing target processing based on the corrected first similarity information includes:

[0088] Step S51, normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs.

[0089] In this step, after normalization, the sum of the probabilities corresponding to the similarities in the same row is equal to 1. Specifically, for any similarity in the modified first similarity information, the similarity and the similarity in the row to which the similarity belongs can be used as inputs of the preset softmax function to generate the probability corresponding to the similarity. Among them, the preset softmax function can map each similarity in the row to between zero and positive infinity, and then calculate the ratio between the mapping value corresponding to the similarity and the sum of the mapping values ​​corresponding to each similarity in the row to which it belongs, so as to achieve normalization and obtain the probability corresponding to the similarity. In this way, the target probability information can be obtained by performing normalization based on the softmax function, thereby ensuring processing efficiency to a certain extent.

[0090] Figure 3 is a schematic diagram of a processing process provided by an embodiment of the present disclosure, such as Figure 3 As shown, "softmax, dim = 1" means that the first similarity matrix 01 is subjected to softmax processing in the vertical direction to obtain the first probability matrix 02, and then the first similarity matrix 01 and the first probability matrix 02 are dotted (Dot) to obtain the modified first similarity matrix 03. "softmax, dim = 0" means that the modified first similarity matrix 03 is subjected to softmax processing in the horizontal direction to obtain the target probability matrix 04.

[0091] Further, Figure 4 is another processing process schematic diagram provided by the embodiment of the present disclosure, such as Figure 4 As shown, if the first similarity matrix 11 is directly subjected to softmax processing in the horizontal direction to generate the probability matrix 12, all videos will be matched with the first sentence, resulting in an error. In the disclosed embodiment, the first probability matrix 13 is first generated based on the first similarity matrix 11, and then the modified first similarity matrix 14 is obtained by dot multiplication, and finally the target probability matrix 15 is generated based on the modified first similarity matrix 14. The target probability matrix 15 indicates that the first sentence has the highest probability of matching the first video, the second sentence has the highest probability of matching the second video, and the third sentence has the highest probability of matching the third video. It can be seen that video text retrieval can be achieved more accurately based on the target probability matrix 15.

[0092] Step S52: Determine the output value of the retrieval model according to the target probability information.

[0093] In this step, the target probability information can be directly used as the output value of the retrieval model, wherein the second media information corresponding to the column with the maximum probability in each row of the target probability information is the second media information that matches the first media information corresponding to the row. Alternatively, the second media information that matches each first media information can be determined based on the target probability information, and then the identification information of the first media information and the identification information of the second media information that matches the first media information are used as the output value of the retrieval model. The identification information can be a name, a number, etc.

[0094] In the disclosed embodiment, the corrected first similarity information is normalized to obtain target probability information, and the probability in the target probability information is used to characterize the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs. The output value of the retrieval model is determined based on the target probability information. In this way, the output value of the retrieval model is generated based on the target probability information determined based on the corrected first similarity information, which can make the final retrieval result more accurate to a certain extent, thereby optimizing the retrieval effect.

[0095] Optionally, when the retrieval model has not completed training, the performing target processing based on the corrected first similarity information includes:

[0096] Step S61: Calculate a first loss value according to the corrected first similarity information and the first probability information.

[0097] In the disclosed embodiment, the elements in the corrected first similarity information and the elements in the first probability information can be used as inputs of the first preset loss function to obtain the output of the first preset loss function, thereby obtaining a first loss value. In the processing of the first preset loss function, the corrected first similarity information can be normalized to obtain the target probability information. Then, the first loss value is further calculated based on the elements in the target probability information and the elements in the first probability information. The implementation method for obtaining the target probability information can refer to the aforementioned related description and will not be repeated here. For example, it is assumed that the first media information is a video and the second media information is a text. With L v2t Represents the first loss value, then the first loss value can be expressed as:

[0098]

[0099] Wherein, B represents batch-size, that is, the number of first media information and second media information participating in this round of retrieval. v and s are the corresponding video and text feature representations. l represents a preset constant, represents the elements in the first probability information, exp(l·sim(v i ,si ) in sim(v i ,s i ) represents the mapping value corresponding to the similarity of the ith video to the ith text. represents the sum of the mapping values ​​corresponding to the similarity of the i-th video to each text. sim represents the similarity calculation function, sim(v i ,s i ) represents the similarity in the modified first similarity matrix.

[0100]

[0101]

[0102] Step S62: Determine the loss value of the retrieval model based on the first loss value.

[0103] For example, the first loss value may be directly determined as the loss value of the retrieval model. Of course, it is also possible to obtain the loss value of the retrieval model by only performing calculations based on the first loss value.

[0104] In the disclosed embodiment, the first loss value is calculated based on the corrected first similarity information and the first probability information. Based on the first loss value, the loss value of the retrieval model is determined. In this way, by further introducing prior probability information when calculating the loss value, a richer and more useful calculation basis can be provided for the loss function, which can improve the accuracy of the calculated loss value to a certain extent, and then improve the optimization effect of model training to a certain extent, ensure the retrieval recall ability of the trained retrieval model, and thus ensure the retrieval effect of cross-modal retrieval.

[0105] Optionally, the following steps may also be performed in the embodiment of the present disclosure:

[0106] Step S71: Determine second probability information that the first media information matches the second media information according to second similarity information between the second media information and the first media information.

[0107] In this step, the first media information matching the second media information can be retrieved from the first media information, and the similarity between each second media information and each first media information can be calculated based on the characteristics of each second media information and the characteristics of each first media information to form the second similarity information, and the second similarity information can be a second similarity matrix in the form of a matrix. Further, the similarity can be normalized according to the similarity in the column to which the similarity belongs to generate the probability corresponding to the similarity, thereby generating the second probability information, and the second probability information can be a second probability matrix in the form of a matrix.

[0108] Step S72: correct the second similarity information based on the second probability information, and calculate a second loss value based on the corrected second similarity information and the second probability information.

[0109] In this step, a dot product may be performed based on the second similarity matrix and the second probability matrix to perform matrix correction to obtain a corrected second similarity matrix.

[0110] Furthermore, the elements in the modified second similarity matrix and the elements in the second probability matrix can be used as inputs of the second preset loss function to obtain the output of the second preset loss function, thereby obtaining a second loss value. In the processing of the second preset loss function, the modified second similarity matrix can be normalized to obtain a third probability matrix. Then, the second loss value is further calculated based on the elements in the third probability matrix and the elements in the second probability matrix. The implementation method for obtaining the third probability matrix can refer to the aforementioned related description and will not be repeated here. For example, with L t2v Represents the second loss value, then the second loss value can be expressed as:

[0111]

[0112] Among them, exp(l·sim(v i， s i ) where sim(vi, si) represents the mapping value corresponding to the similarity of the ith text to the ith video. Represents the sum of the mapping values ​​corresponding to the similarity of the i-th text to each video. represents the elements in the second probability matrix.

[0113]

[0114] Accordingly, the above-mentioned operation of determining the loss value of the retrieval model based on the first loss value may specifically include: step S73, calculating the loss value of the retrieval model based on the first loss value and the second loss value. In this step, the sum of the first loss value and the second loss value may be determined as the loss value of the retrieval model. Furthermore, the model parameters of the retrieval model may be adjusted based on the loss value, and the training optimization of the retrieval model may be achieved through deep learning training. In the disclosed embodiment, the second probability information of the match between the first media information and the second media information is determined based on the second similarity information of the second media information relative to the first media information. The second similarity information is corrected based on the second probability information, and the second loss value is calculated based on the corrected second similarity information and the second probability information. The first loss value and the second loss value are determined as the loss value of the retrieval model. In this way, the ability of the retrieval model to retrieve the first media information based on the second media information can be trained, and the processing capability of the trained retrieval model can be further ensured, thereby ensuring the retrieval effect of cross-modal retrieval.

[0115] In a related technology, in the video text retrieval framework, the usual retrieval tasks are often directly applied, such as the triplet loss function (triplet loss) and the symmetric cross-entropy loss function (symmetric cross-entropyloss). In the disclosed embodiment, a customized design is made for the characteristics of video text retrieval, and the prior probability matrix is ​​further calculated vertically and introduced into the loss function calculation. In this way, some outliers can be filtered out and the confidence points can be sharpened. In addition, this solution does not require additional computing resources and storage resource overhead. When it is implemented specifically, it only needs to add code for introducing the prior probability matrix. The changes brought about are small, the implementation cost is low, and at the same time the model recall rate can be further improved. For example, Figure 5 It is another processing process schematic diagram provided by the present disclosure, such as Figure 5 As shown, for the similarity matrix 31, the retrieval model trained using the conventional loss function has more discrete points in the final probability distribution 32, while the retrieval model trained using the loss function of the present invention has fewer discrete points in the final probability distribution 33, which is more consistent with the actual diagonal distribution.

[0116] Figure 6 is a block diagram of a media information processing device provided by an embodiment of the present disclosure, such as Figure 6 As shown, the device 20 may include:

[0117] The extraction module 201 is configured to extract features of first media information and features of second media information based on a retrieval model; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information;

[0118] A first determination module 202 is configured to determine first similarity information of the first media information relative to the second media information based on a feature of the first media information and a feature of the second media information;

[0119] A second determination module 203 is configured to determine first probability information that the second media information matches the first media information according to the first similarity information;

[0120] The execution module 204 is configured to modify the first similarity information based on the first probability information, and perform target processing based on the modified first similarity information; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model.

[0121] In summary, the media information processing device provided by the embodiment of the present disclosure extracts the features of the first media information and the features of the second media information respectively based on the retrieval model; the first media information is information in video mode or text mode, and the second media information is different from the first media information in mode. Based on the features of the first media information and the features of the second media information, the first similarity information of the first media information relative to the second media information is determined. According to the first similarity information, the first probability information of the match between the second media information and the first media information is determined. The first similarity information is corrected based on the first probability information, and the target processing is performed based on the corrected first similarity information; the target processing includes generating the output of the retrieval model and / or determining the loss value of the retrieval model. In this way, the first probability information of the degree of match between the two is measured from the perspective of retrieving the first media information from the second media information, and the first similarity information is further corrected first, which can make the corrected first similarity information more accurate to a certain extent, and then improve the accuracy of subsequent target processing, thereby improving the accuracy of cross-modal retrieval and ensuring the retrieval effect of cross-modal retrieval.

[0122] Optionally, the first similarity information is a first similarity matrix, and the first probability information is a first probability matrix; a row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row relative to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column relative to other first media information; the second determination module 203 is specifically configured as follows:

[0123] For any similarity in the first similarity matrix, a probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs;

[0124] The first probability matrix is ​​generated according to the probabilities.

[0125] Optionally, the second determining module 203 is further configured to:

[0126] The similarity is normalized according to the similarities in the column to which the similarity belongs, so as to generate a probability corresponding to the similarity.

[0127] Optionally, the execution module 204 is specifically configured to:

[0128] A dot product is performed based on the first similarity matrix and the first probability matrix to perform matrix correction.

[0129] Optionally, when the retrieval model has been trained, the execution module 204 is further configured to:

[0130] Normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs;

[0131] An output value of the retrieval model is determined according to the target probability information.

[0132] Optionally, when the retrieval model has not completed training, the execution module 204 is further configured to:

[0133] Calculating a first loss value according to the corrected first similarity information and the first probability information;

[0134] Based on the first loss value, a loss value of the retrieval model is determined.

[0135] Optionally, the device 20 further includes:

[0136] a third determination module, configured to determine second probability information that the first media information matches the second media information based on second similarity information between the second media information and the first media information;

[0137] a calculation module, configured to modify the second similarity information based on the second probability information, and calculate a second loss value based on the modified second similarity information and the second probability information;

[0138] The execution module 204 is further specifically configured to calculate the loss value of the retrieval model based on the first loss value and the second loss value.

[0139] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0140] According to an embodiment of the present disclosure, an electronic device is provided, comprising: a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement the steps in the media information processing method in any of the above embodiments when executing.

[0141] According to an embodiment of the present disclosure, a storage medium is further provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the steps in the media information processing method in any of the above embodiments.

[0142] According to one embodiment of the present disclosure, a computer program product is also provided, which includes readable program instructions. When the readable program instructions are executed by a processor of an electronic device, the electronic device can execute the steps in the media information processing method in any of the above embodiments.

[0143] Figure 7 700 is a block diagram of a device for media information processing according to an exemplary embodiment. For example, the device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0144] Reference Figure 7 , the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0145] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned media information processing method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0146] The memory 704 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0147] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0148] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0149] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0150] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0151] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0152] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0153] In an exemplary embodiment, the device 700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned media information processing method.

[0154] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the device 700 to complete the above-mentioned media information processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0155] Figure 8800 is a block diagram of a device for media information processing according to an exemplary embodiment. For example, the device 800 may be provided as a server. Figure 8 The apparatus 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions executable by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned media information processing method.

[0156] The device 800 may also include a power supply component 826 configured to perform power management of the device 800, a wired or wireless network interface 850 configured to connect the device 800 to a network, and an input / output (I / O) interface 858. The device 800 may operate based on an operating system stored in the memory 832, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0157] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0158] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A media information processing method, characterized in that: The method comprises: Extracting features of first media information and features of second media information based on the retrieval model; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information; Determining first similarity information of the first media information relative to the second media information based on a feature of the first media information and a feature of the second media information; Determining first probability information of matching the second media information with the first media information according to the first similarity information; modifying the first similarity information based on the first probability information, and performing target processing based on the modified first similarity information; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model; The first similarity information is a first similarity matrix, and the first probability information is a first probability matrix; a row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row relative to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column relative to other first media information; determining the first probability information that the second media information matches the first media information according to the first similarity information includes: For any similarity in the first similarity matrix, a probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs; The first probability matrix is ​​generated according to the probabilities.

2. The method according to claim 1, characterized in that The calculating the probability corresponding to the similarity according to the similarity in the column to which the similarity belongs includes: The similarity is normalized according to the similarities in the column to which the similarity belongs, so as to generate a probability corresponding to the similarity.

3. The method according to claim 1, characterized in that: The modifying the first similarity information based on the first probability information includes: A correction is performed by performing a point multiplication based on the first similarity matrix and the first probability matrix.

4. The method according to claim 1, characterized in that In the case where the retrieval model has been trained, the target processing is performed based on the corrected first similarity information, including: Normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs; An output value of the retrieval model is determined according to the target probability information.

5. The method according to claim 1, characterized in that In the case that the retrieval model has not completed training, the target processing is performed based on the corrected first similarity information, including: Calculating a first loss value according to the corrected first similarity information and the first probability information; Based on the first loss value, a loss value of the retrieval model is determined.

6. The method according to claim 5, characterized in that The method further comprises: Determining second probability information that the first media information matches the second media information according to second similarity information between the second media information and the first media information; modifying the second similarity information based on the second probability information, and calculating a second loss value based on the modified second similarity information and the second probability information; Determining the loss value of the retrieval model based on the first loss value includes: calculating the loss value of the retrieval model based on the first loss value and the second loss value.

7. A media information processing device, characterized in that: The device comprises: an extraction module configured to extract features of first media information and features of second media information respectively based on a retrieval model; the first media information is information in a video mode or a text mode, and the second media information is in a different mode from the first media information; A first determining module is configured to determine first similarity information of the first media information relative to the second media information based on a feature of the first media information and a feature of the second media information; a second determination module, configured to determine first probability information that the second media information matches the first media information according to the first similarity information; an execution module, configured to modify the first similarity information based on the first probability information, and perform target processing based on the modified first similarity information; the target processing includes generating an output of the retrieval model and / or determining a loss value of the retrieval model; The first similarity information is a first similarity matrix, and the first probability information is a first probability matrix; a row of similarity in the first similarity matrix is ​​used to characterize the similarity of the first media information corresponding to the row with respect to other second media information, and a column of similarity in the first similarity matrix is ​​used to characterize the similarity of a second media information corresponding to the column with respect to other first media information; the second determination module is specifically configured as follows: For any similarity in the first similarity matrix, a probability corresponding to the similarity is calculated according to the similarity in the column to which the similarity belongs; the probability is used to represent the probability that the second media information corresponding to the column to which the similarity belongs matches the first media information corresponding to the row to which the similarity belongs; The first probability matrix is ​​generated according to the probabilities.

8. The device according to claim 7, characterized in that The second determining module is further configured to: The similarity is normalized according to the similarities in the column to which the similarity belongs, so as to generate a probability corresponding to the similarity.

9. The device according to claim 7, characterized in that The execution module is specifically configured as follows: A dot product is performed based on the first similarity matrix and the first probability matrix to perform matrix correction.

10. The device according to claim 7, characterized in that When the retrieval model has been trained, the execution module is further configured to: Normalizing the corrected first similarity information to obtain target probability information; the probability in the target probability information is used to represent the probability that the first media information corresponding to the row to which the probability belongs matches the second media information corresponding to the column to which the probability belongs; An output value of the retrieval model is determined according to the target probability information.

11. The device according to claim 7, characterized in that In the case that the retrieval model has not completed training, the execution module is further configured to: Calculating a first loss value according to the corrected first similarity information and the first probability information; Based on the first loss value, a loss value of the retrieval model is determined.

12. The device according to claim 11, characterized in that The device also includes: a third determination module, configured to determine second probability information that the first media information matches the second media information based on second similarity information between the second media information and the first media information; a calculation module, configured to modify the second similarity information based on the second probability information, and calculate a second loss value based on the modified second similarity information and the second probability information; The execution module is further specifically configured to calculate the loss value of the retrieval model based on the first loss value and the second loss value.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

14. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 6.

15. A computer program product, characterized in that The computer program product comprises readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 6.

Images