A model training method and device, an article representation method and device, an electronic device, and a medium

Abstract

Embodiments of the present application disclose a model training, an article representation method, device, electronic equipment and medium. The model training method comprises: obtaining a training text representation and a training audio-visual representation of a training article, and a pre-built original article representation model; inputting the training text representation and the training audio-visual representation into an audio-visual purification module to purify the training audio-visual representation based on the training text representation through the audio-visual purification module to obtain and output a purified audio-visual representation; inputting the purified audio-visual representation into an article representation module to process the purified audio-visual representation through the article representation module to obtain and output an article representation of the training article; and performing loss calculation based on the obtained article representations of multiple training articles to adjust parameters in the original article representation model to train a target article representation model. The technical solution of the embodiments of the present application can realize accurate determination of the article representation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training, item representation method, apparatus, electronic device and medium. Background Technology

[0002] Item matching algorithms are used in many scenarios. Generally speaking, an item matching algorithm includes the following two aspects: first, an item representation algorithm, which is used to abstract the features of items to obtain item representations; second, calculating the similarity between the item representations of two items to determine whether the two items match.

[0003] In the process of realizing this invention, the inventors discovered the following technical problems in the prior art: the existing item representation algorithm cannot obtain accurate item representation, which makes the item representation capability insufficient and thus affects the item matching performance. This needs to be improved. Summary of the Invention

[0004] This invention provides a model training method, an object representation method, an apparatus, an electronic device, and a medium to achieve accurate determination of object representation.

[0005] According to one aspect of the present invention, a model training method is provided, which may include:

[0006] Obtain the training text representation and training audio-visual representation of the training items, and the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module;

[0007] The training text representation and training audio-visual representation are input into the audio-visual purification module. The audio-visual purification module purifies the training audio-visual representation based on the training text representation, and then outputs the purified audio-visual representation.

[0008] The purified audio-visual representation is input into the item representation module, so that the purified audio-visual representation is processed by the item representation module to obtain and output the item representation of the training item;

[0009] Loss is calculated based on the item representations of multiple training items to adjust the parameters in the original item representation model, thereby training the target item representation model.

[0010] According to another aspect of the present invention, a method for representing an article is provided, which may include:

[0011] The application text representation and application audio-visual representation of the application item are obtained, as well as the target item representation model trained according to the model training method described in any embodiment of the present invention;

[0012] The applied text representation and applied audio-visual representation are input into the target item representation model, and the item representation of the applied item is obtained based on the output of the target item representation model.

[0013] According to another aspect of the present invention, a model training apparatus is provided, which may include:

[0014] The first model acquisition module is used to acquire the training text representation and training audio-visual representation of the training items, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module.

[0015] The audio-visual representation purification output module is used to input the training text representation and the training audio-visual representation into the audio-visual purification module, so that the audio-visual purification module can purify the training audio-visual representation based on the training text representation, and obtain and output the purified audio-visual representation.

[0016] The item representation output module is used to input the purified audio-visual representation into the item representation module, so that the item representation module can process the purified audio-visual representation to obtain and output the item representation of the training item;

[0017] The model training module is used to calculate the loss based on the item representations of multiple training items, so as to adjust the parameters in the original item representation model and train the target item representation model.

[0018] According to another aspect of the present invention, an article display device is provided, which may include:

[0019] The second model acquisition module is used to acquire the application text representation and application audio-visual representation of the application item, and the target item representation model trained according to the model training method described in any embodiment of the present invention;

[0020] The item representation module is used to input the application text representation and application audio-visual representation into the target item representation model, and obtain the item representation of the application item based on the output of the target item representation model.

[0021] According to another aspect of the present invention, an electronic device is provided, which may include:

[0022] At least one processor; and

[0023] A memory that is communicatively connected to at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by at least one processor, such that when the at least one processor executes the program, it implements the model training method or item representation method provided in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, having stored thereon computer instructions that cause a processor to execute and implement the model training method or item representation method provided in any embodiment of the present invention.

[0026] The technical solution of this invention involves acquiring training text representations and training audio-visual representations of training items, as well as a pre-constructed original item representation model. This original item representation model includes an audio-visual purification module and an item representation module. The training text representations and training audio-visual representations are input into the audio-visual purification module to purify the training audio-visual representations based on the training text representations, resulting in and outputting purified audio-visual representations. The purified audio-visual representations are then input into the item representation module to process them, resulting in and outputting item representations of the training items. Loss calculations are performed based on the obtained item representations of multiple training items to adjust the parameters in the original item representation model, thereby training a target item representation model. This technical solution, by using less noisy training text representations to purify noisy training audio-visual representations, effectively ensures the accuracy of the obtained item representations. Then, model training based on these item representations effectively guarantees the target item representation model's ability in item representation, thus ensuring subsequent item matching performance.

[0027] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a model training method provided according to an embodiment of the present invention;

[0030] Figure 2 This is a flowchart of another model training method provided according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of another model training method provided according to an embodiment of the present invention;

[0032] Figure 4This is a schematic diagram of the structure of a model example in another model training method provided according to an embodiment of the present invention;

[0033] Figure 5 This is a flowchart of an article representation method provided according to an embodiment of the present invention;

[0034] Figure 6 This is a structural block diagram of a model training device provided according to an embodiment of the present invention;

[0035] Figure 7 This is a structural block diagram of an article display device according to an embodiment of the present invention;

[0036] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the model training method or the item representation method of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The same applies to "target," "original," etc., and will not be repeated here. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Figure 1 This is a flowchart of a model training method provided in an embodiment of the present invention. This embodiment is applicable to situations where a target item representation model is trained. The method can be executed by the model training device provided in this embodiment of the present invention. This device can be implemented in software and / or hardware, and can be integrated into an electronic device, which can be various user terminals or servers.

[0040] See Figure 1 The method of this invention specifically includes the following steps:

[0041] S110. Obtain the training text representation and training audio-visual representation of the training item, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module.

[0042] Here, training items can be understood as items used for model training. Training text representation can be understood as the textual representation of the training items; specifically, it can be a representation obtained by processing the textual information of the training items, which may include at least one of the following: title, description, tags, and comments. Training audio-visual representation can be understood as the audio and / or visual representation of the training items; specifically, it can be a representation obtained by processing the audio-visual information of the training items, which may include at least one of the following: audio, video, and images. It should be noted that the representations mentioned above can also be called vectors, vector representations, features, or feature representations, etc., without specific limitations here.

[0043] The original item representation model can be understood as a pre-built and trained model used to determine the item representation of the training item. It can include at least an audio-visual purification module and an item representation module. The audio-visual purification module can be understood as a module used to purify (i.e. denoise) the training audio-visual representation, and the item representation module can be understood as a module used to obtain the item representation based on the purified audio-visual representation output by the audio-visual purification module.

[0044] S120. Input the training text representation and the training audio-visual representation into the audio-visual purification module, so that the audio-visual purification module can purify the training audio-visual representation based on the training text representation, and obtain and output the purified audio-visual representation.

[0045] In practice, it was found that audio-visual information contains more noise than text information. That is, compared with text training representation, training audio-visual representation contains more noise. It should be noted that this noise will directly affect the accuracy of the subsequent item representation.

[0046] Therefore, in order to ensure the accuracy of the item representation, the training text representation and the training audio-visual representation are input into the audio-visual purification module, which then performs the following steps: based on the training text representation, the training audio-visual representation is purified, that is, the noise in the training audio-visual representation is removed, thereby obtaining the purified audio-visual representation (i.e., the purified training audio-visual representation), and the purified audio-visual representation is output.

[0047] S130. Input the purified audio-visual representation into the item representation module, so that the purified audio-visual representation can be processed by the item representation module to obtain and output the item representation of the training item.

[0048] The purified audio-visual representation obtained through the above steps has little or no noise, so it can be used to obtain the object representation, thereby ensuring the accuracy of the object representation.

[0049] Specifically, the purified audio-visual representation is input into the item representation module, which then performs the following steps: processes the purified audio-visual representation to obtain and output the item representation of the training item. In practical applications, optionally, only the purified audio-visual representation can be input into the item representation module, thus processing only the purified audio-visual representation to obtain the item representation; alternatively, both the text training representation and the purified audio-visual representation can be input into the item representation module, and then processed to obtain the item representation; of course, other methods can also be used to combine the purified audio-visual representation to obtain the item representation, without specific limitations here.

[0050] S140. Based on the item representations of the multiple training items, calculate the loss to adjust the parameters in the original item representation model and train the target item representation model.

[0051] In model training, multiple training items are typically involved. For each of these training items, the aforementioned steps are applied to process it, resulting in an item representation. After obtaining the item representation for each training item, a pre-defined learning task can be performed to train the original item representation model. Specifically,

[0052] Loss is calculated based on the item representations of multiple training items to obtain the loss value. It should be noted that the specific implementation method of loss calculation can be determined according to the preset learning task. Furthermore, the parameters in the original item representation model are adjusted according to the loss value so that after multiple rounds of parameter tuning, the target item representation model can be trained. This target item representation model can be understood as the original item representation model that has been trained.

[0053] The technical solution of this invention involves acquiring training text representations and training audio-visual representations of training items, as well as a pre-constructed original item representation model. This original item representation model includes an audio-visual purification module and an item representation module. The training text representations and training audio-visual representations are input into the audio-visual purification module to purify the training audio-visual representations based on the training text representations, resulting in and outputting purified audio-visual representations. The purified audio-visual representations are then input into the item representation module to process them, resulting in and outputting item representations of the training items. Loss calculations are performed based on the obtained item representations of multiple training items to adjust the parameters in the original item representation model, thereby training a target item representation model. This technical solution, by using less noisy training text representations to purify noisy training audio-visual representations, effectively ensures the accuracy of the obtained item representations. Then, model training based on these item representations effectively guarantees the target item representation model's ability in item representation, thus ensuring subsequent item matching performance.

[0054] Based on this, an optional technical solution is provided, in which the item representation module includes a first fusion encoding module, which inputs the purified audio-visual representation into the item representation module, so that the purified audio-visual representation is processed by the item representation module to obtain and output the item representation of the training item, including:

[0055] The purified audio-visual representation and the training text representation are input into the first fusion encoding module, so that the purified audio-visual representation and the training text representation are fused and encoded by the first fusion encoding module to obtain and output the item representation of the training item.

[0056] The first fusion encoding module can be understood as a module used to fuse and encode the purified audio-visual representation and the training text representation to obtain the item representation. The purified audio-visual representation and the training text representation are input into the first fusion encoding module, which then fuses and encodes them to obtain and output the item representation of the training item. In practical applications, the above fusion encoding process can optionally be implemented using at least one of average pooling, max pooling, and concatenation operations; of course, it can also be implemented using other operations, without specific limitations here.

[0057] The above technical solution, by processing the relatively accurate text training representation and the purified audio-visual representation together, results in an item representation that not only has high accuracy but also good richness due to the inclusion of multimodal information, further improving the target item representation model's ability in item representation.

[0058] Another optional technical solution involves training the audio-visual representation, which includes training both audio and video representations. Based on the trained text representation, the trained audio-visual representation is refined to obtain and output the refined audio-visual representation, including:

[0059] Based on the training text representation, the training audio representation is purified to obtain and output the purified audio representation; and the training video representation is purified to obtain and output the purified video representation.

[0060] Correspondingly, the item representation module includes a second fusion encoding module, which inputs the purified audio-visual representation into the item representation module so that the purified audio-visual representation can be processed by the item representation module to obtain and output the item representation of the training item, including:

[0061] The purified audio representation and the purified video representation are input into the second fusion encoding module, so that the purified audio representation and the purified video representation are fused and encoded by the second fusion encoding module to obtain and output the item representation of the training item.

[0062] Here, the training audio representation can be understood as the audio representation of the training item, and the training video representation can be understood as the video representation of the training item. Based on this, the audio-visual purification module purifies the representations in both modalities respectively, that is, purifying the training audio representation to obtain the purified audio representation, and purifying the training video representation to obtain the purified video representation.

[0063] The second fusion encoding module can be understood as a module used to fuse and encode the purified audio representation and the purified video representation to obtain the item representation. Based on this, the purified audio representation and the purified video representation can be input into the second fusion encoding module, which then fuses and encodes the two modal representations to obtain the item representation. In practical applications, optionally, the above fusion encoding process can be implemented using at least one of the operations such as average pooling, max pooling, and concatenation; of course, it can also be implemented using other operations, without specific limitations here.

[0064] The above technical solution, by processing the relatively accurate purified audio representation and purified video representation together, results in an item representation that not only has high accuracy but also good richness due to the inclusion of multimodal information, further improving the target item representation model's ability in item representation.

[0065] Another optional technical solution involves calculating the loss based on the item representations of multiple training items to adjust the parameters in the original item representation model, including:

[0066] For the current item, the first item similar to the current item, and the second item dissimilar to the current item among the multiple training items obtained, determine the first similarity between the current item and the first item, and the second similarity between the current item and the second item respectively;

[0067] Loss is calculated based on the first and second similarity scores to adjust the parameters in the original item representation model.

[0068] Here, "current item" can be understood as the training item currently being processed among multiple training items; "first item" can be understood as a training item similar to the current item among multiple training items; and "second item" can be understood as a training item dissimilar to the current item among multiple training items. In practical applications, optionally, some or all of the multiple training items can be used as the current item; optionally, if there are at least two training items similar to the current item among multiple training items, some or all of these training items can be used as the first item; further optionally, the second item can be the same as or different from the first item, which will not be elaborated here.

[0069] Building upon this foundation, to improve item matching performance, a first similarity is determined between the current item and the first item, measuring their relevance. A second similarity is also determined between the current item and the second item, measuring their relevance. Then, loss calculation, or model learning, is performed based on these two similarities. This yields a target item representation model with better item representation capabilities, ensuring the accuracy of subsequent item matching applications using this model.

[0070] Figure 2 This is a flowchart of another model training method provided in this embodiment of the invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, based on the training text representation, the training audio-visual representation is purified to obtain and output the purified audio-visual representation, which may include: obtaining the gating attention weights of the training audio-visual representation according to the training text representation; purifying the training audio-visual representation according to the gating attention weights to obtain the purified audio-visual representation, and outputting the purified audio-visual representation. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0071] See Figure 2 The method in this embodiment may specifically include the following steps:

[0072] S210. Obtain the training text representation and training audio-visual representation of the training item, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module.

[0073] S220. Input the training text representation and the training audio-visual representation into the audio-visual purification module. The audio-visual purification module obtains the gating attention weights of the training audio-visual representation based on the training text representation, and purifies the training audio-visual representation based on the gating attention weights to obtain and output the purified audio-visual representation.

[0074] Specifically, by using the training text representation and a gating attention mechanism, the gating attention weights of the training audio-visual representation can be obtained. These gating attention weights can be used to characterize the probability that each part of the information in the training audio-visual representation is valid information. Then, using the gating attention weights, the training audio-visual representation is refined to obtain a refined audio-visual representation, which is then output.

[0075] S230. Input the purified audio-visual representation into the item representation module, so that the purified audio-visual representation can be processed by the item representation module to obtain and output the item representation of the training item.

[0076] S240. Based on the item representations of the multiple training items, calculate the loss to adjust the parameters in the original item representation model and train the target item representation model.

[0077] The technical solution of this invention utilizes the training text representation and combines it with a gating attention mechanism to obtain the gating attention weights of the training audio-visual representation. Then, the gating attention weights are used to effectively purify the training audio-visual representation, thereby ensuring the accuracy of the subsequent item representations obtained based on it.

[0078] Based on this, an optional technical solution obtains the gating attention weights for training audio-visual representations according to the training text representation, including:

[0079] Obtain the text weights corresponding to the current training text representation, and the audio-visual weights corresponding to the current training audio-visual representation; based on the training text representation, text weights, training audio-visual representation, and audio-visual weights, obtain the gating attention weights of the training audio-visual representation;

[0080] The parameters in the original item representation model include at least text weights and audio-visual weights.

[0081] In this context, text weights can be understood as the weights used to calculate the gating attention weights corresponding to the current training text representation, and audio-visual weights can be understood as the weights used to calculate the gating attention weights corresponding to the current training audio-visual representation. It's important to note that the quality of the text and audio-visual weight settings directly affects the accuracy of the gating attention weight calculation. Therefore, after calculating the gating attention weights based on the two representations and two weights, subsequent steps can be performed based on these gating attention weights to adjust the text and audio-visual weights according to the calculated loss value. Through multiple rounds of adjustment, the accuracy of the calculated gating attention weights can be ensured, thereby guaranteeing the effectiveness of the audio-visual representation purification process.

[0082] Figure 3 This is a flowchart of another model training method provided in this embodiment of the invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, the number of training text representations is at least two and the number of training audio-visual representations is at least two. The original item representation model further includes a text encoding module and an audio-visual encoding module. The above model training method further includes: inputting at least two training text representations into the text encoding module to encode the at least two training text representations to obtain and output the context text representations corresponding to the at least two training text representations respectively; inputting at least two training audio-visual representations into the audio-visual encoding module to encode the at least two training audio-visual representations to obtain and output the comprehensive audio-visual representations corresponding to the at least two training audio-visual representations respectively; inputting the training text representations and training audio-visual representations into the audio-visual purification module to purify the training audio-visual representations based on the training text representations to obtain and output the purified audio-visual representations, including: inputting at least two context text representations and at least two comprehensive audio-visual representations into the audio-visual purification module to purify the at least two comprehensive audio-visual representations based on the at least two context text representations to obtain and output the purified audio-visual representations. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0083] See Figure 3 The method in this embodiment may specifically include the following steps:

[0084] S310. Obtain the training text representation and training audio-visual representation of the training item, as well as the pre-built original item representation model, wherein the original item representation model includes a text encoding module, an audio-visual encoding module, an audio-visual purification module, and an item representation module.

[0085] The training text representations can be two or more, and these training text representations are independent of each other and have no correlation. Similarly, the training audio-visual representations can be two or more, and these training audio-visual representations are independent of each other and have no correlation.

[0086] A text encoding module can be understood as a module that encodes at least two training text representations to establish a correlation between them. Similarly, an audio-visual encoding module can be understood as a module that encodes at least two training audio-visual representations to establish a correlation between them. In practical applications, optionally, the text encoding module and / or the audio-visual encoding module can be implemented using a transformer encoder network.

[0087] S320. Input at least two training text representations into the text encoding module, so as to encode the at least two training text representations through the text encoding module, and obtain and output the context text representations corresponding to the at least two training text representations respectively.

[0088] In this process, at least two training text representations are input into a text encoding module, which then performs the following steps: encoding the at least two training text representations so that they learn contextual information, thereby obtaining a contextual text representation corresponding to each of the at least two training text representations. In other words, each training text representation corresponds to its own contextual text representation, and compared to the initial training text representations, these contextual text representations are correlated and can reflect the textual information of the training item as a whole, thus improving the textual representation capability of the training item.

[0089] S330. Input at least two training audio-visual representations into the audio-visual encoding module, so as to encode the at least two training audio-visual representations through the audio-visual encoding module, and obtain and output the comprehensive audio-visual representations corresponding to the at least two training audio-visual representations respectively.

[0090] In this process, at least two training audio-visual representations are input into an audio-visual encoding module. The module then performs the following steps: encoding the at least two training audio-visual representations to enable interactive learning between them, resulting in a comprehensive audio-visual representation corresponding to each training audio-visual representation. In other words, each training audio-visual representation corresponds to its own comprehensive audio-visual representation. Compared to the initial training audio-visual representations, these comprehensive audio-visual representations are correlated, integrating overall audio-visual information, and can comprehensively reflect the audio-visual information, thereby improving the ability to represent training items in terms of audio-visual aspects.

[0091] S340. Input at least two context text representations and at least two integrated audio-visual representations into the audio-visual purification module, so that the audio-visual purification module can purify the at least two integrated audio-visual representations based on the at least two context text representations, and obtain and output the purified audio-visual representations.

[0092] The process involves inputting at least two contextual text representations and at least two integrated audio-visual representations into an audio-visual purification module. The module then performs the following steps: purifying the integrated audio-visual representations based on the contextual text representations. This can be achieved by fusing the contextual text representations to obtain a final text representation, and then purifying the integrated audio-visual representations based on the final text representation; or by fusing the integrated audio-visual representations to obtain a final audio-visual representation, and then purifying the final audio-visual representation based on the final text representation; or by purifying the final audio-visual representations based on the contextual text representations; and so on. The number of purified audio-visual representations obtained can be one, two, or more, depending on the specific operation of the audio-visual purification module, and is not specifically limited here. Optionally, if the number of purified audio-visual representations is two or more, these purified audio-visual representations can be fused, and the fusion result can be used as the purified audio-visual representation, thereby improving the convenience of the item representation module.

[0093] S350. Input the purified audio-visual representation into the item representation module, so that the purified audio-visual representation can be processed by the item representation module to obtain and output the item representation of the training item.

[0094] S360. Based on the item representations of multiple training items, calculate the loss to adjust the parameters in the original item representation model and train the target item representation model.

[0095] The technical solution of this invention provides a contextual text representation obtained by encoding at least two training text representations, which can reflect the text information as a whole. Furthermore, a comprehensive audio-visual representation obtained by encoding at least two training audio-visual representations can comprehensively reflect audio-visual information. Therefore, refining these comprehensive audio-visual representations based on these contextual text representations helps to improve the refinement effect, thereby further ensuring the accuracy of the item representation obtained based on this.

[0096] Based on this, an alternative technical solution, the above model training method, further includes:

[0097] At least two contextual text representations are fused to obtain the final text representation;

[0098] At least two contextual text representations and at least two integrated audio-visual representations are input into the audio-visual purification module, so that the audio-visual purification module can purify the at least two integrated audio-visual representations based on the at least two contextual text representations, including:

[0099] The final text representation and at least two integrated audio-visual representations are input into the audio-visual purification module, so that the audio-visual purification module can purify the at least two integrated audio-visual representations based on the final text representation.

[0100] Compared to contextual text representation, the final text representation better reflects the overall text information. Therefore, by using the final text representation, at least two integrated audio-visual representations can be purified, thereby further improving the purification effect.

[0101] Another alternative technical solution, the above model training method, also includes:

[0102] Obtain the text information and audio-visual information of the training text, perform word segmentation on the text information to obtain at least two text segments, and extract frames from the audio-visual information to obtain at least two audio-visual frames;

[0103] Determine the word representation of each text word in at least two text word segments respectively, and use the word representation as the training text representation to obtain at least two training text representations;

[0104] Determine the frame representation of each of the at least two audio-visual frames, and use the frame representation as the training audio-visual representation to obtain at least two training audio-visual representations.

[0105] This process involves segmenting the text information into words to obtain at least two text segments. Then, the segmentation representation of each of these segments is determined, and these representations are used as training text representations, resulting in at least two training text representations. In practical applications, optionally, a dictionary can be used to look up the word vector for each text segment, and then the word vector can be used as the segmentation representation. The resulting at least two segmentation representations can also be referred to as a word vector sequence.

[0106] By extracting frames from the audio-visual information, at least two audio-visual frames are obtained. Then, the frame representation of each audio-visual frame is determined, and this frame representation is used as a training audio-visual representation, thus obtaining at least two training audio-visual representations. In practical applications, optionally, the above frame extraction process can be implemented using the ffmpeg tool. Alternatively, when the audio-visual information is video, an image model can be used to model the video frames, thereby obtaining the frame representation of each video frame. Considering the application scenarios that may be involved in the embodiments of this invention, this image model can be constructed based on a convolutional neural network; here, the ResNet model is used as an example for representing video frames. When the audio-visual information is audio, a sound model can be used to model the audio frames, thereby obtaining the frame representation of each audio frame. Considering the application scenarios that may be involved in the embodiments of this invention, the Vggish model is used as an example for modeling audio frames.

[0107] The above technical solution achieves the effect of accurately determining the training text representation and the training audio-visual representation.

[0108] To better understand the various technical solutions described above, a specific example is provided below. For each training item, its textual, video, and audio information are obtained based on its Stock Keeping Unit (SKU). Then, the item representation is obtained by representing and fusing information from these three modalities. It should be noted that textual modality information (i.e., textual data) is more accurate than information from other modalities due to its richer and more explicit attribute information. Therefore, this example proposes a textual modality selection mechanism to leverage the dominant role of the textual modality in the feature fusion process, resulting in an item representation that is both rich and accurate.

[0109] For details, see Figure 4 Perform the following two steps:

[0110] (1) Item representation method based on text modality selection mechanism and hierarchical fusion mechanism

[0111] After acquiring video, audio, and text information of the training items, the information is processed to represent them in the model. Then, a multimodal network is built for modeling and learning is performed based on a preset task.

[0112] a) Video features

[0113] First, the video frames are modeled using an image model to obtain the frame representation v of each video frame. l At this point, the frame representation of the L video frames is through v1-v LThis is illustrated. Based on this, the frame representations of L video frames are fused using the video encoding module (i.e., the Vision Encoder module) to obtain a comprehensive video representation that incorporates overall video information. At this point, the comprehensive video representation of L video frames can be achieved through... To illustrate.

[0114] b) Audio characteristics

[0115] Similar to the video processing process, the audio frames are first modeled using the Vggish model to obtain the frame representation 'a' for each audio frame. n At this point, the frame representation of N audio frames is achieved through a1-a N This is illustrated. Based on this, in order to obtain a comprehensive audio representation of the audio frame... Similarly, by modeling the audio frames through the audio encoding module (i.e., the Audio Encoder module), the comprehensive audio representation of N audio frames can be obtained through... To illustrate.

[0116] c) Text Features

[0117] By segmenting the text information into words and querying the word vector t for each segmented word, the text information can be processed. m At this point, the word vectors of the M text segments can be obtained through t1-t M For illustration, these can be called word vector sequences. Then, the word vector sequences are input into the text encoding module (i.e., the Text Encoder module) to encode the text information, obtaining a contextual text representation that incorporates contextual information. At this point, the contextual text representation of the M text segments can be obtained through... These can be referred to as text sequence features.

[0118] Furthermore, the text sequence features are fused to obtain the final text representation h. t Here, average pooling and max pooling are used to achieve the fusion process of text sequence features, as detailed below:

[0119]

[0120] d) Constructing item representations through text modality selection and hierarchical fusion mechanisms

[0121] First, use the final text representation h t The gating attention weights for video and audio are obtained separately using a gating attention mechanism, as shown in the following formula:

[0122]

[0123]

[0124] Among them, W v W represents the video weights. a W represents audio weights. t Characterizing text weights, The gating attention weights that characterize the video. Gated attention weights characterize audio.

[0125] Next, we use gated attention weights to refine the video and audio, i.e., denoise them, as follows:

[0126]

[0127]

[0128] in, The characterization and purification video indicates that... Characterize purified audio representation.

[0129] Furthermore, the final video representation h is obtained through fusion pooling operations. v and the final audio representation h a The formula is as follows:

[0130]

[0131]

[0132] Finally, the video, audio, and text information are fused together using the Fusion Encoder module to obtain the item representation h of the training item. f The formula is as follows:

[0133] h f =concat[mean_pool([h v ,h t ,h a ]),max_pool([h v ,h t ,h a ])]

[0134] It should be noted that, as explained above, the text modality selection mechanism in this example can be understood as selecting effective information from the video and audio modalities through text modality, and the hierarchical fusion mechanism can be understood as... Figure 4 The diagram illustrates the layer-by-layer fusion of multimodal information. Taking the video side as an example, it starts from v l arrive arrive to h vThen h f Each time a layer is merged upwards, more information is incorporated, thus ensuring the final item representation h is accurate. f The accuracy.

[0135] (2) Item matching correlation calculation

[0136] To improve the item matching effect, the item representation h output in (1) is used. f Item relevance is measured for multiple training items. Specifically, cosine similarity is used to measure the similarity between two items, and margin loss is used for model learning, as follows:

[0137]

[0138] in, and The model is trained based on the item representations of the current item (i.e. the item to be compared), the first item similar to the item to be compared, and the second item dissimilar to the item to be compared, to obtain the target item representation model.

[0139] Figure 5 This is a flowchart of an item representation method provided in an embodiment of the present invention. This embodiment is applicable to situations where the item representation of an applied item is determined. The method can be executed by the item representation device provided in this embodiment of the present invention. This device can be implemented in software and / or hardware, and can be integrated into an electronic device, which can be various user terminals or servers.

[0140] See Figure 5 The method of this invention specifically includes the following steps:

[0141] S410. Obtain the application text representation and application audio-visual representation of the application item, and the target item representation model trained according to the model training method described in any embodiment of the present invention.

[0142] In this context, "application item" can be understood as the item involved in the model application process. "Application text representation" can be understood as the textual representation of the application item, specifically, it can be a representation obtained by processing the textual information of the application item, such as at least one of titles, descriptions, tags, and comments. "Application audiovisual representation" can be understood as the audio and / or visual representation of the application item, specifically, it can be a representation obtained by processing the audiovisual information of the application item, such as at least one of audio, video, and images. "Target item representation model" can be understood as a model trained according to any of the above technical solutions, used to determine the item representation.

[0143] S420. Input the applied text representation and applied audio-visual representation into the target item representation model, and obtain the item representation of the applied item based on the output of the target item representation model.

[0144] The text representation and audio-visual representation are input into the target item representation model, and the item representation of the applied item can be obtained based on the output of the target item representation model.

[0145] The technical solution of this invention involves inputting the application text representation and application audio-visual representation of the applied item into the target item representation model, thereby obtaining an accurate and high-quality item representation of the applied item based on the output of the target item representation model.

[0146] Based on this, an alternative technical solution, the above model training method, further includes:

[0147] Retrieve item representations from multiple candidate items;

[0148] For each candidate item among multiple candidate items, determine a third similarity between the item representation of the applied item and the item representation of that candidate item;

[0149] Based on the third similarity of multiple candidate items, the target item that matches the application item is determined from the multiple candidate items.

[0150] In this context, the item representations of candidate items are predetermined. In practical applications, optionally, for each candidate item among multiple candidate items, the item representation of the candidate item can be predetermined, and then a representation pool (or vector pool) can be constructed based on these item representations. Thus, operations related to item representations, such as item matching, can be performed based on these item representations in the representation pool.

[0151] The system obtains the item representations of multiple candidate items, and then determines the third similarity between the item representation of the applied item and the item representation of each candidate item. Based on these third similarities, the system determines the target item that matches the applied item from the multiple candidate items. For example, the candidate items with the third similarity in the Top N can be used as the target items, where N is a positive integer. This achieves the item matching effect.

[0152] In practical applications, the above-mentioned item matching scheme can be applied in multiple scenarios. For example, on e-commerce platforms, when marketing copy needs to be generated for a certain app's product but the product information for that app is insufficient, the above-mentioned item matching scheme can find a target product that matches the app's product. Based on the target product's information, more product information can be added to the app's product, thereby improving the quality of the generated copy and increasing the app's product exposure. As another example, when a user wants to learn about an app's product but the product information for that app is insufficient, the above-mentioned item matching scheme can find a target product that matches the app's product, thus providing the user with more product information based on the target product.

[0153] Figure 6 This is a structural block diagram of a model training apparatus provided in an embodiment of the present invention. This apparatus is used to execute the model training method provided in any of the above embodiments. This apparatus and the model training methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the model training apparatus can be found in the embodiments of the above model training methods. See also... Figure 6 The device may specifically include: a model first acquisition module 510, a purified audio-visual representation output module 520, an item representation output module 530, and a model training module 540.

[0154] The first model acquisition module 510 is used to acquire the training text representation and training audio-visual representation of the training items, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module.

[0155] The audio-visual representation purification output module 520 is used to input the training text representation and the training audio-visual representation into the audio-visual purification module, so that the audio-visual purification module can purify the training audio-visual representation based on the training text representation to obtain and output the purified audio-visual representation.

[0156] The item representation output module 530 is used to input the purified audio-visual representation into the item representation module so that the purified audio-visual representation can be processed by the item representation module to obtain and output the item representation of the training item.

[0157] The model training module 540 is used to perform loss calculation based on the item representations of multiple training items to adjust the parameters in the original item representation model and train the target item representation model.

[0158] Optional, the refined audio-visual representation output module 520 includes:

[0159] The gating attention weights submodule is used to obtain the gating attention weights of the training audio-visual representation based on the training text representation;

[0160] The audio-visual representation purification output submodule is used to purify the training audio-visual representation according to the gating attention weights, obtain the purified audio-visual representation, and output the purified audio-visual representation.

[0161] Based on this, optional sub-modules for obtaining gating attention weights include:

[0162] The audio-visual weight acquisition unit is used to acquire the text weights corresponding to the current training text representation, and the audio-visual weights corresponding to the current training audio-visual representation.

[0163] The gating attention weight acquisition unit is used to obtain the gating attention weights of the training audio-visual representation based on the training text representation, text weights, training audio-visual representation, and audio-visual weights.

[0164] The parameters in the original item representation model include at least text weights and audio-visual weights.

[0165] Optionally, the item representation module includes a first fusion encoding module and an item representation output module 530, specifically used for:

[0166] The purified audio-visual representation and the training text representation are input into the first fusion encoding module, so that the purified audio-visual representation and the training text representation are fused and encoded by the first fusion encoding module to obtain and output the item representation of the training item.

[0167] Optionally, training the audio-visual representation includes training the audio representation and training the video representation. The audio-visual representation refinement output module 520 includes:

[0168] The purified video representation output submodule is used to purify the training audio representation based on the training text representation, obtain and output the purified audio representation, and to purify the training video representation, obtain and output the purified video representation.

[0169] The item representation module includes a second fusion encoding module and an item representation output module 530, which are specifically used for:

[0170] The purified audio representation and the purified video representation are input into the second fusion encoding module, so that the purified audio representation and the purified video representation are fused and encoded by the second fusion encoding module to obtain and output the item representation of the training item.

[0171] Optionally, the number of training text representations is at least two and the number of training audio-visual representations is at least two. The original item representation model also includes a text encoding module and an audio-visual encoding module.

[0172] The aforementioned model training device also includes:

[0173] The context text representation output module is used to input at least two training text representations into the text encoding module, so that the text encoding module encodes the at least two training text representations to obtain and output the context text representations corresponding to the at least two training text representations respectively;

[0174] The integrated audio-visual representation output module is used to input at least two training audio-visual representations into the audio-visual encoding module, so that the audio-visual encoding module encodes the at least two training audio-visual representations to obtain and output the integrated audio-visual representations corresponding to the at least two training audio-visual representations respectively.

[0175] The audio-visual representation output module 520 includes:

[0176] The audio-visual representation purification output submodule is used to input at least two context text representations and at least two comprehensive audio-visual representations into the audio-visual purification module, so that the audio-visual purification module can purify the at least two comprehensive audio-visual representations based on the at least two context text representations, and obtain and output the purified audio-visual representation.

[0177] In addition, optionally, the above-mentioned model training device further includes:

[0178] The final text representation module is used to fuse at least two context text representations to obtain the final text representation;

[0179] The audio-visual representation output submodule includes:

[0180] The integrated audio-visual representation purification unit is used to input the final text representation and at least two integrated audio-visual representations into the audio-visual purification module, so that the audio-visual purification module can purify the at least two integrated audio-visual representations based on the final text representation.

[0181] Alternatively, the aforementioned model training device may further include:

[0182] The audio-visual frame acquisition module is used to acquire the text information and audio-visual information of the training text, and to perform word segmentation on the text information to obtain at least two text segments, and to extract frames from the audio-visual information to obtain at least two audio-visual frames.

[0183] The training text representation acquisition module is used to determine the word representation of each text word in at least two text word segments, and use the word representation as the training text representation to obtain at least two training text representations;

[0184] The training audio-visual representation acquisition module is used to determine the frame representation of each audio-visual frame in at least two audio-visual frames, and use the frame representation as the training audio-visual representation to obtain at least two training audio-visual representations.

[0185] Optional, model training module 540, including:

[0186] The second similarity determination submodule is used to determine the first similarity between the current item and the first item, and the second similarity between the current item and the second item, respectively, for the current item, the first item similar to the current item, and the second item dissimilar to the current item among the multiple training items obtained;

[0187] The parameter tuning submodule is used to perform loss calculations based on the first and second similarities to adjust the parameters in the original item representation model.

[0188] The model training apparatus provided in this embodiment of the invention acquires training text representations and training audio-visual representations of training items through a first model acquisition module, and a pre-constructed original item representation model, which includes an audio-visual purification module and an item representation module. The training text representations and training audio-visual representations are input into the audio-visual purification module through a purified audio-visual representation output module, so that the audio-visual representations are purified based on the training text representations to obtain and output purified audio-visual representations. The purified audio-visual representations are input into the item representation module through the item representation output module, so that the purified audio-visual representations are processed to obtain and output item representations of the training items. The model training module performs loss calculations based on the obtained item representations of multiple training items to adjust the parameters in the original item representation model and train a target item representation model. The aforementioned device refines the noisy training audio-visual representation by using a less noisy training text representation, thereby effectively ensuring the accuracy of the item representation obtained. Then, the model is trained based on the item representation, which effectively ensures the capability of the target item representation model in item representation, thus guaranteeing the performance in subsequent item matching.

[0189] The model training apparatus provided in this embodiment of the invention can execute the model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0190] It is worth noting that in the embodiments of the above-mentioned model training device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0191] Figure 7This is a structural block diagram of an article display device provided in an embodiment of the present invention. This device is used to execute the article display method provided in any of the above embodiments. This device and the article display methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the article display device can be found in the embodiments of the above article display methods. See also... Figure 7 The device may specifically include: a second model acquisition module 610 and an item representation acquisition module 620.

[0192] The second model acquisition module 610 is used to acquire the application text representation and application audio-visual representation of the application item, as well as the target item representation model trained according to the model training method described in any embodiment of the present invention.

[0193] The item representation module 620 is used to input the application text representation and the application audio-visual representation into the target item representation model, and obtain the item representation of the application item based on the output of the target item representation model.

[0194] Optionally, the above-mentioned article display device further includes:

[0195] The item representation acquisition module is used to acquire item representations of multiple candidate items;

[0196] The third similarity determination module is used to determine the third similarity between the item representation of the applied item and the item representation of the candidate item for each of the multiple candidate items.

[0197] The target item determination module is used to determine the target item that matches the application item from multiple candidate items based on the third similarity corresponding to each candidate item.

[0198] The item representation device provided in this embodiment of the invention uses a second model acquisition module and an item representation acquisition module to cooperate with each other to input the application text representation and application audio-visual representation of the applied item into the target item representation model. Based on the output of the target item representation model, an accurate and high-quality item representation of the applied item can be obtained.

[0199] The article display device provided in the embodiments of the present invention can execute the article display method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0200] It is worth noting that in the embodiments of the above-mentioned item display device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0201] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0202] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0203] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0204] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as model training methods or item representation methods.

[0205] In some embodiments, the model training method or item representation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the model training method or item representation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the model training method or item representation method by any other suitable means (e.g., by means of firmware).

[0206] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0207] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0208] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0209] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0210] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0211] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0212] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0213] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A model training method, characterized in that, include: The training text representation and training audio-visual representation of the training items are obtained, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module; The training text representation and the training audio-visual representation are input into the audio-visual purification module, so that the training audio-visual representation is purified based on the training text representation by the audio-visual purification module, and the purified audio-visual representation is obtained and output. The purified audio-visual representation is input into the item representation module, so that the purified audio-visual representation is processed by the item representation module to obtain and output the item representation of the training item; Loss is calculated based on the item representations of the obtained training items to adjust the parameters in the original item representation model and train the target item representation model. The step of refining the training audio-visual representation based on the training text representation to obtain and output the refined audio-visual representation includes: Obtain the text weights currently corresponding to the training text representation, and the audio-visual weights currently corresponding to the training audio-visual representation, wherein the parameters in the original item representation model include at least the text weights and the audio-visual weights; Based on the training text representation, the text weights, the training audio-visual representation, and the audio-visual weights, the gating attention weights of the training audio-visual representation are obtained; Based on the gating attention weights, the trained audio-visual representation is purified to obtain a purified audio-visual representation, and the purified audio-visual representation is output.

2. The method according to claim 1, characterized in that, The item representation module includes a first fusion encoding module. The step of inputting the purified audio-visual representation into the item representation module, processing the purified audio-visual representation through the item representation module to obtain and output the item representation of the training item, includes: The purified audio-visual representation and the training text representation are input into the first fusion encoding module, so that the purified audio-visual representation and the training text representation are fused and encoded by the first fusion encoding module to obtain and output the item representation of the training item.

3. The method according to claim 1, characterized in that, The trained audio-visual representation includes a trained audio representation and a trained video representation. The step of refining the trained audio-visual representation based on the trained text representation to obtain and output the refined audio-visual representation further includes: Based on the training text representation, the training audio representation is purified to obtain and output the purified audio representation; and the training video representation is purified to obtain and output the purified video representation. The item representation module includes a second fusion encoding module. The step of inputting the purified audio-visual representation into the item representation module, processing the purified audio-visual representation through the item representation module, and obtaining and outputting the item representation of the training item includes: The purified audio representation and the purified video representation are input into the second fusion encoding module, so that the purified audio representation and the purified video representation are fused and encoded by the second fusion encoding module to obtain and output the item representation of the training item.

4. The method according to claim 1, characterized in that, The number of training text representations is at least two and the number of training audio-visual representations is at least two. The original item representation model also includes a text encoding module and an audio-visual encoding module. The method further includes: The at least two training text representations are input into the text encoding module to encode the at least two training text representations, thereby obtaining and outputting the context text representations corresponding to the at least two training text representations respectively. The at least two training audio-visual representations are input into the audio-visual encoding module, so that the at least two training audio-visual representations are encoded by the audio-visual encoding module to obtain and output the comprehensive audio-visual representations corresponding to the at least two training audio-visual representations respectively. The step of inputting the training text representation and the training audio-visual representation into the audio-visual purification module, so as to purify the training audio-visual representation based on the training text representation through the audio-visual purification module, and obtain and output the purified audio-visual representation, further includes: The at least two context text representations and the at least two integrated audio-visual representations are input into the audio-visual purification module, so that the at least two integrated audio-visual representations are purified based on the at least two context text representations to obtain and output the purified audio-visual representations.

5. The method according to claim 4, characterized in that, Also includes: The at least two contextual text representations are fused to obtain the final text representation; The at least two context text representations and the at least two integrated audio-visual representations are input into the audio-visual purification module, so that the at least two integrated audio-visual representations are purified based on the at least two context text representations, including: The final text representation and the at least two integrated audio-visual representations are input into the audio-visual purification module, so that the audio-visual purification module can purify the at least two integrated audio-visual representations based on the final text representation.

6. The method according to claim 4, characterized in that, Also includes: The text information and audio-visual information of the training text are obtained, and the text information is segmented to obtain at least two text segments. The audio-visual information is then extracted to obtain at least two audio-visual frames. Determine the word representation of each of the at least two text word segments, and use the word representation as the training text representation to obtain the at least two training text representations; The frame representation of each of the at least two audio-visual frames is determined, and the frame representation is used as the training audio-visual representation to obtain the at least two training audio-visual representations.

7. The method according to claim 1, characterized in that, Loss is calculated based on the item representations of multiple training items to adjust the parameters in the original item representation model, including: For the current item, a first item similar to the current item, and a second item dissimilar to the current item among the multiple training items obtained, a first similarity between the current item and the first item, and a second similarity between the current item and the second item are determined respectively; Loss is calculated based on the first similarity and the second similarity to adjust the parameters in the original item representation model.

8. A method for representing an item, characterized in that, include: The application text representation and application audio-visual representation of the application item are obtained, and the target item representation model is trained according to the model training method of any one of claims 1-7. The application text representation and the application audio-visual representation are input into the target item representation model, and the item representation of the application item is obtained based on the output of the target item representation model.

9. The method according to claim 8, characterized in that, Also includes: Retrieve item representations of multiple candidate items; For each of the plurality of candidate items, a third similarity is determined between the item representation of the applied item and the item representation of the candidate item; Based on the third similarity corresponding to each of the multiple candidate items, a target item matching the application item is determined from the multiple candidate items.

10. A model training device, characterized in that, include: The first model acquisition module is used to acquire the training text representation and training audio-visual representation of the training items, as well as the pre-built original item representation model, wherein the original item representation model includes an audio-visual purification module and an item representation module; The audio-visual representation purification output module is used to input the training text representation and the training audio-visual representation into the audio-visual purification module, so that the audio-visual purification module purifies the training audio-visual representation based on the training text representation to obtain and output the purified audio-visual representation. The item representation output module is used to input the purified audio-visual representation into the item representation module, so that the purified audio-visual representation is processed by the item representation module to obtain and output the item representation of the training item; The model training module is used to perform loss calculation based on the item representations of multiple training items, so as to adjust the parameters in the original item representation model and train the target item representation model. The purified audio-visual representation output module includes: The audio-visual weight acquisition unit is used to acquire the text weight currently corresponding to the training text representation, and the audio-visual weight currently corresponding to the training audio-visual representation, wherein the parameters in the original item representation model include at least the text weight and the audio-visual weight; The gated attention weight obtaining unit is used to obtain the gated attention weight of the training audio-visual representation based on the training text representation, the text weight, the training audio-visual representation, and the audio-visual weight; The audio-visual representation purification output submodule is used to purify the training audio-visual representation according to the gating attention weights, obtain the purified audio-visual representation, and output the purified audio-visual representation.

11. An article display device, characterized in that, include: The second model acquisition module is used to acquire the application text representation and application audio-visual representation of the application item, and the target item representation model trained according to the model training method of any one of claims 1-7; The item representation module is used to input the application text representation and the application audio-visual representation into the target item representation model, and obtain the item representation of the application item based on the output of the target item representation model.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to cause the at least one processor to perform the model training method as described in any one of claims 1-7, or the item representation method as described in claim 8 or 9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the model training method as described in any one of claims 1-7, or the item representation method as described in claim 8 or 9.

Images