Visual analysis methods and devices

Abstract

This application relates to the field of computer vision and provides a visual analysis method and apparatus. The visual analysis method includes determining a first text feature based on first text data, and determining a first visual feature based on first visual data. The first visual feature includes multiple levels of visual features of the first visual data, wherein the visual features at each level correspond to different scales. Cross-level attention interaction is performed on the first text feature and the first visual feature to obtain a first interaction feature. If the similarity between the first text feature and the first interaction feature is greater than a first threshold, the first text data is determined as semantic information of the first visual data. The embodiments of this application can accurately extract semantic information from visual data.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a visual analysis method and apparatus. Background Technology

[0002] Video semantic understanding has a wider range of applications than image understanding, especially with the current popularity of short videos, which are increasingly being used by industries. For example, video search results offer advantages over text and image results, such as being more intuitive, having a lower reading threshold, providing a more immersive experience, containing more information, being easier to protect original content, and being easier to monetize. Video search has become a new trend in the search field.

[0003] Related technologies employ a single-scale approach, involving the interaction of visual and textual features, followed by masking and prediction for training, ultimately applying to downstream cross-modal tasks. Current large-scale model pre-training largely originates from a masking-prediction pre-training paradigm similar to Bidirectional Encoder Representations from Transformer (BERT) models, which can utilize massive unlabeled corpus datasets for training. Gradually, this paradigm has been adopted by the computer vision field, introducing additional loss constraints. However, when applied to video-text pre-training, this paradigm first disrupts the video input, then predicts the complete video after the interaction of video and textual features. This training method has certain problems. For example, in the multimodal interaction of visual and textual information, the video disruption easily introduces noise, limiting generalization in video semantic understanding. Furthermore, using single-scale video features to interact with textual features fails to capture fine-grained visual features effectively. Summary of the Invention

[0004] In view of this, a visual analysis method and device are proposed, which can accurately extract the semantic information of visual data.

[0005] In a first aspect, embodiments of this application provide a visual analysis method, comprising:

[0006] A first text feature is determined based on first text data, and a first visual feature is determined based on first visual data, wherein the first visual feature includes multiple levels of visual features of the first visual data, and the scales corresponding to the visual features at each level are different.

[0007] Cross-level attention interaction is performed on the first text feature and the first visual feature to obtain the first interaction feature;

[0008] If the similarity between the first text feature and the first interaction feature is greater than a first threshold, the first text data is determined as the semantic information of the first visual data.

[0009] The embodiments provided in this application utilize visual features and text features at multiple scales to interact. The visual features can simultaneously cover coarse-grained and fine-grained features, which helps to significantly improve the interaction capabilities between different modalities, thereby enabling accurate extraction of semantic information from visual data. In addition, the use of multiple sets of attention interactions can reduce the use of graphics processing unit (GPU) memory.

[0010] In a first possible implementation of the visual analysis method, the cross-level attention interaction includes attention interactions between two adjacent levels, wherein, in the attention interactions between the two adjacent levels, the input of the attention interaction of the next level is the output of the attention interaction of the previous level.

[0011] The embodiments provided in this application set up cross-level attention interactions, inputting the output of the attention interaction of the previous level into the attention interaction of the next level. At the same time, they utilize deep semantic features and shallow fine-grained features for text interaction, which helps to improve the output accuracy of attention interactions, so that the first interaction features contain the semantic features of the first visual data.

[0012] In a second possible implementation of the visual analysis method, the visual features of the first visual data at multiple levels include at least a first feature and a second feature, wherein the levels corresponding to the first feature and the second feature are adjacent and the level corresponding to the first feature is higher than the level corresponding to the second feature, and the cross-level attention interaction between the first text feature and the first visual feature includes:

[0013] Attention interaction is performed on the first feature and the third feature to obtain a fourth feature. Wherein, if the level corresponding to the first feature is the highest level among the multiple levels, the third feature is the first text feature. If the level corresponding to the first feature is not the highest level among the multiple levels, the third feature is the output of the level above the level corresponding to the first feature.

[0014] The second feature and the fourth feature are interacted with by attention to obtain the fifth feature;

[0015] If the level corresponding to the second feature is the lowest level among the plurality of levels, the fifth feature is used as the first interaction feature;

[0016] If the level corresponding to the second feature is not the lowest level among the multiple levels, the fifth feature is input into the next level of the level corresponding to the second feature for attention interaction until the first interaction feature is obtained.

[0017] The embodiments provided in this application utilize the interaction between the first text features and the highest-level visual features, and input the result of the interaction into the attention interaction of the next level, which is progressive and helps to output more reliable features.

[0018] In a third possible implementation of the visual analysis method, determining the first text feature based on the first text data and determining the first visual feature based on the first visual data include:

[0019] The first text data is input into the text encoder, and the first text feature is output.

[0020] The first visual data is input into the visual encoder, and the first visual feature is output.

[0021] The embodiments provided in this application can determine more accurate text features and visual features through trained text encoders and visual encoders, which is beneficial for subsequent interaction with text features and visual features and improves interaction capabilities.

[0022] In a fourth possible implementation of the visual analysis method, the method further includes:

[0023] The text encoder and the visual encoder are trained based on a training set, which includes multiple training sample pairs, each of which includes second text data and second visual data.

[0024] The embodiments provided in this application train the text encoder and the visual encoder using paired related visual and text data, which helps to obtain more accurate text encoders and visual encoders, thereby helping to determine more accurate prediction results in practical applications.

[0025] In a fifth possible implementation of the visual analysis method, training the text encoder and the visual encoder based on the training set includes:

[0026] For any training sample pair in the training set:

[0027] The second text data and the second visual data in the training sample pair are respectively input into the text encoder and the visual encoder to be trained, and the second text feature and the second visual feature are output. The second visual feature includes multiple levels of visual features of the second visual data.

[0028] Cross-level attention interaction is performed on the second text feature and the second visual feature to obtain the second interaction feature;

[0029] Based on the second text features and the second interaction features, determine the first loss corresponding to the training sample pair;

[0030] Based on the first loss corresponding to each training pair, the parameters of the text encoder and the parameters of the visual encoder are adjusted.

[0031] The embodiments provided in this application utilize multiple training samples to train the text encoder and visual encoder, and use the loss between the predicted result and the ideal result to back-update the parameters of the text encoder and visual encoder, which helps to obtain a more accurate text encoder and visual encoder, thereby obtaining a better visual analysis model.

[0032] Among the six possible implementations of the visual analysis method, the method further includes:

[0033] For any training sample pair in the training set, feature comparison learning is performed based on the second visual feature to obtain the second loss corresponding to the training sample pair;

[0034] The parameters of the text encoder and the visual encoder are adjusted based on the second loss corresponding to each training sample pair.

[0035] The embodiments provided in this application use a self-supervised learning mode of contrastive learning, which helps to avoid destroying visual information and affecting the attention interaction process during attention interaction. It also uses the results of contrastive learning to adjust the parameters of the visual analysis model, which is conducive to obtaining a more accurate visual analysis model. Thus, the visual analysis model can be used to perform better visual analysis.

[0036] In the seventh possible implementation of the visual analysis method, the step of performing feature contrast learning based on the second visual features to obtain the second loss corresponding to the training sample pair includes:

[0037] For any one of the multiple levels:

[0038] Based on the visual features of the second visual data at the level, determine the global and local features at the level;

[0039] Based on the global and local features of the level, determine the second loss corresponding to the level;

[0040] The second loss corresponding to each training sample pair is determined based on the second loss corresponding to each level.

[0041] The embodiments provided in this application not only utilize global features, but also fully explore local features between multiple levels, which can improve the effect of self-supervised learning. Furthermore, by using a multi-level global-local contrastive learning approach, visual representations can be made more robust, thereby further enhancing the interaction between different modal information.

[0042] In the eighth possible implementation of the visual analysis method, determining the second loss corresponding to the level based on the global and local features of the level includes:

[0043] From the local features at the level, determine the positive and negative features corresponding to the global features at the level;

[0044] Based on the global features, the positive features, and the negative features, the second loss corresponding to the level is determined.

[0045] The embodiments provided in this application utilize a multi-level global-local contrastive learning approach, which not only further enhances the robustness of visual modalities, thereby promoting cross-attention interaction between subsequent visual features and text features, but also uses contrastive learning to supervise the training of the visual analysis model, thereby obtaining a high-precision visual analysis model, which is beneficial for subsequent visual analysis work.

[0046] In a ninth possible implementation of the visual analysis method, the method further includes:

[0047] Upon receiving a visual data query request, the first text data is obtained from the visual data query request, and the first visual data is obtained from the visual database to be queried.

[0048] If the first text data is semantic information of the first visual data, then return the first visual data;

[0049] And / or,

[0050] Upon receiving a visual data annotation request, the first text data is obtained from the candidate tag library, and the first visual data is obtained from the visual data annotation request;

[0051] If the first text data is semantic information of the first visual data, return the first text data;

[0052] And / or,

[0053] Upon receiving a visual data question-and-answer request, the question information and the first visual data are obtained from the visual data question-and-answer request, and the first text data is determined based on the question information.

[0054] The embodiments provided in this application can not only meet users' video retrieval needs and help users obtain more intuitive and three-dimensional video results compared to text and image results; they can also tag massive amounts of unlabeled videos, and the tag information can simultaneously contain fine-grained features and coarse-grained features, which is beneficial for applications such as video recommendation; they can also provide corresponding answers to question information based on visual data, and can be widely used in the field of machine vision, such as 3D digital human question answering.

[0055] Secondly, embodiments of this application provide a visual analysis device, comprising:

[0056] A feature generation unit is configured to determine a first text feature based on first text data, and to determine a first visual feature based on first visual data, wherein the first visual feature includes multiple levels of visual features of the first visual data, wherein the scales corresponding to the visual features at each level are different.

[0057] An attention interaction unit is configured to perform cross-level attention interaction on the first text feature and the first visual feature to obtain a first interaction feature.

[0058] A semantic output unit is configured to determine the first text data as semantic information of the first visual data when the similarity between the first text feature and the first interaction feature is greater than a first threshold.

[0059] The embodiments provided in this application utilize visual features and text features at multiple scales to interact. The visual features can simultaneously cover coarse-grained and fine-grained features, which helps to significantly improve the interaction capabilities between different modalities, thereby enabling accurate extraction of semantic information from visual data. In addition, the use of multiple sets of attention interactions can reduce the use of graphics processing unit (GPU) memory.

[0060] In a first possible implementation of the visual analysis device, the cross-level attention interaction includes attention interactions between two adjacent levels, wherein, in the attention interactions between the two adjacent levels, the input of the attention interaction of the next level is the output of the attention interaction of the previous level.

[0061] The embodiments provided in this application set up cross-level attention interactions, inputting the output of the attention interaction of the previous level into the attention interaction of the next level. At the same time, they utilize deep semantic features and shallow fine-grained features for text interaction, which helps to improve the output accuracy of attention interactions, so that the first interaction features contain the semantic features of the first visual data.

[0062] In a second possible implementation of the visual analysis device, the visual features of the first visual data at multiple levels include at least a first feature and a second feature, wherein the levels corresponding to the first feature and the second feature are adjacent and the level corresponding to the first feature is higher than the level corresponding to the second feature, and the attention interaction unit is further configured to:

[0063] Attention interaction is performed on the first feature and the third feature to obtain a fourth feature. Wherein, if the level corresponding to the first feature is the highest level among the multiple levels, the third feature is the first text feature. If the level corresponding to the first feature is not the highest level among the multiple levels, the third feature is the output of the level above the level corresponding to the first feature.

[0064] The second feature and the fourth feature are interacted with by attention to obtain the fifth feature;

[0065] If the level corresponding to the second feature is the lowest level among the plurality of levels, the fifth feature is used as the first interaction feature;

[0066] If the level corresponding to the second feature is not the lowest level among the multiple levels, the fifth feature is input into the next level of the level corresponding to the second feature for attention interaction until the first interaction feature is obtained.

[0067] The embodiments provided in this application utilize the interaction between the first text features and the highest-level visual features, and input the result of the interaction into the attention interaction of the next level, which is progressive and helps to output more reliable features.

[0068] In a third possible implementation of the visual analysis device, the feature generation unit is further configured as follows:

[0069] The first text data is input into the text encoder, and the first text feature is output.

[0070] The first visual data is input into the visual encoder, and the first visual feature is output.

[0071] The embodiments provided in this application can determine more accurate text features and visual features through trained text encoders and visual encoders, which is beneficial for subsequent interaction with text features and visual features and improves interaction capabilities.

[0072] In a fourth possible implementation of the visual analysis device, the device further includes:

[0073] A training unit is configured to train the text encoder and the visual encoder based on a training set, the training set including multiple training sample pairs, each training sample pair including second text data and second visual data.

[0074] The embodiments provided in this application train the text encoder and the visual encoder using paired related visual and text data, which helps to obtain more accurate text encoders and visual encoders, thereby helping to determine more accurate prediction results in practical applications.

[0075] In a fifth possible implementation of the visual analysis device, the training unit is further configured as follows:

[0076] For any training sample pair in the training set:

[0077] The second text data and the second visual data in the training sample pair are respectively input into the text encoder and the visual encoder to be trained, and the second text feature and the second visual feature are output. The second visual feature includes multiple levels of visual features of the second visual data.

[0078] Cross-level attention interaction is performed on the second text feature and the second visual feature to obtain the second interaction feature;

[0079] Based on the second text features and the second interaction features, determine the first loss corresponding to the training sample pair;

[0080] Based on the first loss corresponding to each training pair, the parameters of the text encoder and the parameters of the visual encoder are adjusted.

[0081] The embodiments provided in this application utilize multiple training samples to train the text encoder and visual encoder, and use the loss between the predicted result and the ideal result to back-update the parameters of the text encoder and visual encoder, which helps to obtain a more accurate text encoder and visual encoder, thereby obtaining a better visual analysis model.

[0082] In a sixth possible implementation of the visual analysis device, the device further includes:

[0083] A comparison unit is configured to perform feature comparison learning based on the second visual features for any training sample pair in the training set, and obtain a second loss corresponding to the training sample pair.

[0084] An adjustment unit is configured to adjust the parameters of the text encoder and the parameters of the visual encoder based on the second loss corresponding to each training sample pair.

[0085] The embodiments provided in this application use a self-supervised learning mode of contrastive learning, which helps to avoid destroying visual information and affecting the attention interaction process during attention interaction. It also uses the results of contrastive learning to adjust the parameters of the visual analysis model, which is conducive to obtaining a more accurate visual analysis model. Thus, the visual analysis model can be used to perform better visual analysis.

[0086] In a seventh possible implementation of the visual analysis device, the comparison unit is further configured as follows:

[0087] For any one of the multiple levels:

[0088] Based on the visual features of the second visual data at the level, determine the global and local features at the level;

[0089] Based on the global and local features of the level, determine the second loss corresponding to the level;

[0090] The second loss corresponding to each training sample pair is determined based on the second loss corresponding to each level.

[0091] The embodiments provided in this application not only utilize global features, but also fully explore local features between multiple levels, which can improve the effect of self-supervised learning. Furthermore, by using a multi-level global-local contrastive learning approach, visual representations can be made more robust, thereby further enhancing the interaction between different modal information.

[0092] In an eighth possible implementation of the visual analysis device, determining the second loss corresponding to the level based on the global and local features at the level includes:

[0093] From the local features at the level, determine the positive and negative features corresponding to the global features at the level;

[0094] Based on the global features, the positive features, and the negative features, the second loss corresponding to the level is determined.

[0095] The embodiments provided in this application utilize a multi-level global-local contrastive learning approach, which not only further enhances the robustness of visual modalities, thereby promoting cross-attention interaction between subsequent visual features and text features, but also uses contrastive learning to supervise the training of the visual analysis model, thereby obtaining a high-precision visual analysis model, which is beneficial for subsequent visual analysis work.

[0096] In a ninth possible implementation of the visual analysis device, the device further includes:

[0097] The first acquisition unit is configured to, upon receiving a visual data query request, acquire the first text data from the visual data query request and acquire the first visual data from the visual database to be queried.

[0098] A first return unit is configured to return the first visual data when the first text data is semantic information of the first visual data.

[0099] And / or,

[0100] The second acquisition unit is configured to, upon receiving a visual data annotation request, acquire the first text data from a candidate tag library and acquire the first visual data from the visual data annotation request;

[0101] The second return unit is configured to return the first text data when the first text data is semantic information of the first visual data;

[0102] And / or,

[0103] The third acquisition unit is configured to, upon receiving a visual data question-and-answer request, acquire question information and the first visual data from the visual data question-and-answer request, and determine the first text data based on the question information;

[0104] The third return unit is configured to return the first text data when the first text data is semantic information of the first visual data.

[0105] The embodiments provided in this application can not only meet users' video retrieval needs and help users obtain more intuitive and three-dimensional video results compared to text and image results; they can also tag massive amounts of unlabeled videos, and the tag information can simultaneously contain fine-grained features and coarse-grained features, which is beneficial for applications such as video recommendation; they can also provide corresponding answers to question information based on visual data, and can be widely used in the field of machine vision, such as 3D digital human question answering.

[0106] Thirdly, embodiments of this application provide a terminal device, including:

[0107] processor;

[0108] Memory used to store processor-executable instructions;

[0109] The processor is configured to implement the visual analysis method described above when executing the instructions.

[0110] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement one or more of the visual analysis methods described in the first aspect or various possible implementations of the first aspect.

[0111] Fifthly, embodiments of this application provide a computer program product including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in an electronic device, the processor in the electronic device executes one or more of the visual analysis methods described in the first aspect or various possible implementations of the first aspect.

[0112] These and other aspects of this application will become more apparent in the description of the following embodiments(s). Attached Figure Description

[0113] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0114] Figure 1 A flowchart illustrating a visual analysis method according to an embodiment of this application is shown.

[0115] Figure 2 A schematic diagram of attention interaction according to an embodiment of this application is shown.

[0116] Figure 3 A first exemplary schematic diagram of cross-level interaction according to an embodiment of this application is shown.

[0117] Figure 4 A second exemplary schematic diagram of cross-level interaction according to an embodiment of this application is shown.

[0118] Figure 5 A schematic diagram of the structure of a visual analysis model according to an embodiment of this application is shown.

[0119] Figure 6 A schematic diagram of the first training process of a visual analysis model according to an embodiment of this application is shown.

[0120] Figure 7 A schematic diagram of the second training process of the visual analysis model according to an embodiment of this application is shown.

[0121] Figure 8 A first flowchart of a visual analysis method according to an embodiment of this application is shown.

[0122] Figure 9 This diagram illustrates a second flow chart of a visual analysis method according to an embodiment of this application.

[0123] Figure 10 This diagram illustrates a third process flow of a visual analysis method according to an embodiment of this application.

[0124] Figure 11 A schematic diagram of the structure of a visual analysis device provided according to an embodiment of this application is shown.

[0125] Figure 12 A schematic diagram of the structure of a terminal device provided according to an embodiment of this application is shown. Detailed Implementation

[0126] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0127] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0128] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0129] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the technical environment for implementing the technical solutions will be described below.

[0130] Compared to text and images, video information offers advantages such as a lower reading threshold, a more immersive experience, greater information content, easier protection of original content, and easier monetization, making video search a new trend in the search field. Users can obtain the information they need through short video searches, and platforms can attract a large number of users by leveraging short video search. However, short video search relies heavily on high-precision visual analysis models. Most related technologies employ a single-scale interaction approach, which cannot effectively utilize both fine-grained and coarse-grained visual features to interact with text features, resulting in lower model performance. Furthermore, most related technologies use masking for self-supervised learning of the input visual data, which disrupts the interaction between different modalities and is also detrimental to improving model performance. Therefore, identifying a high-performance visual analysis model is of great significance.

[0131] To address the aforementioned technical problems, embodiments of this application provide a visual analysis method and apparatus capable of accurately extracting semantic information from visual data. This semantic information enables various applications, such as video retrieval and video classification. The visual analysis method can be applied to terminal devices, facilitating video retrieval and classification operations for users on these devices.

[0132] The terminal device involved in this application can refer to a device with wireless connectivity, which means that it can connect to other terminal devices via wireless connection methods such as Wi-Fi and Bluetooth. The terminal device of this application can also have wired communication capabilities. The terminal device of this application can be touchscreen, non-touchscreen, or screenless. Touchscreen devices can be controlled by clicking or swiping on the display screen using fingers or styluses. Non-touchscreen devices can connect to input devices such as mice, keyboards, and touch panels to control the terminal device. Screenless devices can be, for example, screenless Bluetooth speakers. For instance, the terminal device of this application can be a smartphone, netbook, tablet computer, laptop computer, wearable electronic device (such as a smart bracelet, smartwatch, etc.), TV, virtual reality device, etc.

[0133] Figure 1 A flowchart of a visual analysis method provided according to an embodiment of this application is shown. The method can be applied to terminal devices. Figure 1 As shown, the method may include:

[0134] Step S101: Determine the first text feature based on the first text data, and determine the first visual feature based on the first visual data.

[0135] Here, the first text data can represent any text data. The first text data can be text information from a preset library, or text information input by the user from outside the preset library. The first text feature can represent the text features corresponding to the first text data. By extracting the features of the first text data, the first text feature can be obtained.

[0136] Here, "first visual data" can represent any single visual data point. First visual data can be a video and / or an image. "First visual features" can represent the visual characteristics corresponding to the first visual data; by extracting the features from the first visual data, the first visual features can be obtained.

[0137] The method can be applied to video retrieval. In a video retrieval scenario, the first text data can be text data output by the user, such as one or more keywords, or one or more sentences; the first visual data can be various visual data in the visual database to be queried, for example, the first visual data can be short videos, long videos, or GIFs.

[0138] The method can be applied to video annotation. In a video annotation scenario, the first text data can be any tag in the candidate tag library, such as keywords, phrases, or sentences; the first visual data can be the video to be annotated.

[0139] The method can be applied to video question answering. In a video question answering scenario, the first text data can be keywords for the question and answer, and the first visual data can be the video in which the question is being asked. For example, the question might be: How many apples are in the video?

[0140] The above is merely an exemplary description of the first text data and the first visual data. The first text data and the first visual data involved in the embodiments of this application may also be other text data and visual data, and the embodiments of this application do not limit them.

[0141] In this embodiment, the first visual feature may include visual features at multiple levels of the first visual data, wherein the scales corresponding to the visual features at each level are different. The multiple levels of visual features may include coarse-grained features and fine-grained features. In one example, the scale corresponding to the visual feature at each level may be related to the image resolution. For example, if the resolution of the first visual data is P, then the first visual feature may include three levels of visual features, with resolutions of 1 / 8*P, 1 / 16*P, and 1 / 32*P, respectively. The visual feature with a resolution of 1 / 8*P can be considered a fine-grained feature, while the visual features with resolutions of 1 / 16*P and 1 / 32*P can be considered coarse-grained features. Of course, the first visual feature may also include more or fewer levels of visual features, for example, it may include visual features with resolutions of 1 / 8*P and 1 / 16*P, or visual features with resolutions of 1 / 4*P, 1 / 8*P, 1 / 16*P, 1 / 32*P, and 1 / 64*P. In this embodiment, there is no limitation on the number of layers included in the first visual feature, nor on the scale corresponding to each layer of visual feature.

[0142] Step S102: Perform cross-level attention interaction on the first text feature and the first visual feature to obtain the first interaction feature.

[0143] In one possible implementation, the first text data is a sentence, and the first visual data is a video. By extracting features from the sentence, the first text features can be obtained; by sampling features from the video, the first visual features can be obtained. Figure 2 A schematic diagram of attention interaction according to an embodiment of this application is shown. Figure 2 As shown, taking the first visual features, which include visual features at a 1 / 8 scale, a 1 / 16 scale, and a 1 / 32 scale, as an example, the first interactive feature is obtained by performing attention interactions between the first text feature and the visual features at each level layer by layer. This achieves cross-level attention interaction between the first text feature and the first visual feature. The cross-level attention interaction process between the first text feature and the first visual feature will be explained in detail later, and will not be repeated here.

[0144] Step S103: If the similarity between the first text feature and the first interaction feature is greater than a first threshold, the first text data is determined as the semantic information of the first visual data.

[0145] The correlation between the first visual data and the first text data can be determined based on the similarity and a first threshold of the first interaction feature and the first text feature. The first threshold can be set based on experience or actual conditions, allowing the judgment based on this threshold to determine whether the first text data can serve as semantic information of the first visual data, without limiting the scope of protection of this application. If the similarity is greater than the first threshold, it indicates a high degree of correlation between the first visual data and the first text data. In this case, the first text data can be identified as the semantic information of the first visual data, meaning the semantic information of the first visual data matches the first text data corresponding to the first text feature. The semantic information can be words or sentences. If the similarity is less than or equal to the first threshold, it indicates a low degree of correlation between the first visual data and the first text data. This means the first text data cannot well reflect the semantic information of the first visual data, or the semantic information of the first visual data differs significantly from the first text data; in short, the first text data and the first visual data do not match well. Therefore, for different application scenarios, new first visual data or new first text data can be selected to re-execute the visual analysis method until the similarity between the obtained first interaction feature and the first text feature is greater than the first threshold. For example, in the current application scenario of video retrieval, a visual analysis method can be performed by selecting new first visual data and the current first text data to obtain first visual data that matches the current first text data (see below for details). As another example, in the current application scenario of video annotation, a visual analysis method can be performed by selecting new first text data and the current first visual data to obtain first text data that matches the current first visual data (see below for details).

[0146] In one possible implementation, the similarity between the first interactive feature and the first text feature can be determined by the Euclidean distance, the cosine of the included angle, or the correlation coefficient. Of course, other methods in related technologies can also be used to determine the similarity between the two, and this application embodiment does not limit this.

[0147] Compared to using visual features and text features at a single scale for interaction, the visual analysis method provided in this application uses visual features and text features at multiple scales for interaction. The visual features can simultaneously cover coarse-grained and fine-grained features, which helps to significantly improve the interaction capabilities between different modalities, thereby enabling accurate extraction of semantic information from visual data. In addition, using multiple sets of attention interactions can reduce the use of graphics processing unit (GPU) memory.

[0148] The following section provides a detailed explanation of the cross-level attention interaction process between the first text feature and the first visual feature.

[0149] In this embodiment of the application, cross-level attention interaction is performed on the first text feature and the first visual feature. The cross-level attention interaction includes attention interaction between two adjacent levels. In the attention interaction between two adjacent levels, the input of the attention interaction of the next level is the output of the attention interaction of the previous level.

[0150] Figure 3 A first exemplary schematic diagram of cross-level interaction provided according to an embodiment of this application is shown. Figure 3 As shown, N levels of visual features (i.e., first visual features) are extracted from the first visual data, specifically including: first-level visual features, second-level visual features, ..., Nth-level visual features, where N is an integer greater than or equal to 2. In the embodiments of this application, cross-level attention interaction can be performed based on these N levels of visual features and the first text features. Specifically, as... Figure 3As shown, attention interactions can be performed on the first text features and the first-level visual features to obtain the output of the first-level attention interaction. This output can then be input into the next level, the second level, for further attention interactions. This process continues until the output of the Nth level attention interaction is obtained. Taking the first-level attention interaction process as an example: a self-attention mechanism can be applied to the first text data to obtain its output. Then, a cross-attention mechanism can be applied to this output and the first-level visual features to obtain its output. This cross-attention mechanism output can then be input into a single-layer or multi-layer feedforward neural network to obtain the first-level attention interaction output. The attention interactions of other levels can refer to the first-level attention interaction process, which will not be elaborated further here.

[0151] It should be noted that the aforementioned first-level visual feature is the highest-level visual feature among multiple levels of visual features, and the aforementioned Nth-level visual feature is the lowest-level visual feature among multiple levels of visual features. Furthermore, in this embodiment, the dimension of the output of attention interaction at each level is consistent with the dimension of the first text feature. Therefore, the dimension of the first interaction feature obtained by performing cross-level attention interaction on the first text feature and the first visual feature is consistent with the dimension of the first text feature.

[0152] The visual analysis method provided in this application sets up cross-level attention interactions, inputting the output of the attention interaction of the previous level into the attention interaction of the next level. At the same time, it uses deep semantic features and shallow fine-grained features for text interaction, which helps to improve the output accuracy of attention interactions, so that the first interaction features contain the semantic features of the first visual data.

[0153] The following section will further illustrate the process of cross-level attention interaction between the first text feature and the first visual feature, using the first and second adjacent features in multiple levels of visual features as examples.

[0154] In one possible implementation, the visual features of the first visual data at multiple levels may include at least a first feature and a second feature, wherein the levels corresponding to the first feature and the second feature are adjacent and the level corresponding to the first feature is higher than the level corresponding to the second feature. In this case, step S102, which performs cross-level attention interaction on the first text feature and the first visual feature to obtain the first interactive feature, may include: performing attention interaction on the first feature and the third feature to obtain a fourth feature, wherein if the level corresponding to the first feature is the highest level among multiple levels, the third feature is the first text feature; if the level corresponding to the first feature is not the highest level among multiple levels, the third feature is the output of the level preceding the level corresponding to the first feature; performing attention interaction on the second feature and the fourth feature to obtain a fifth feature; if the level corresponding to the second feature is the lowest level among multiple levels, the fifth feature is used as the first interactive feature; if the level corresponding to the second feature is not the lowest level among multiple levels, the fifth feature is input into the next level of the level corresponding to the second feature for attention interaction, until the first interactive feature is obtained.

[0155] Figure 4 A second exemplary schematic diagram of cross-level interaction provided according to an embodiment of this application is shown. For example... Figure 4 As shown, attention interactions are applied to the first and third features to obtain the fourth feature; attention interactions are applied to the second and fourth features to obtain the fifth feature. Assume that... Figure 3 As shown, the visual features of the first visual data at multiple levels include the first-level visual features, the second-level visual features, ..., the Nth-level visual features, where N is an integer greater than or equal to 2. N can be set according to the actual situation and is not used to limit the scope of protection of this application. The first-level visual features are the highest-level visual features among the multiple levels of visual features, and the Nth-level visual features are the lowest-level visual features among the multiple levels of visual features.

[0156] In one example, the first feature can be a visual feature of the first level mentioned above, and the second feature can be a visual feature of the second level mentioned above. In this case, the first level corresponding to the first feature is the highest level among N levels, and the third feature is the first text feature. The process of cross-level attention interaction on the first text feature and the first visual feature can include: performing attention interaction on the first feature (i.e., the visual feature of the first level) and the third feature (i.e., the first text feature) to obtain the fourth feature (i.e., the output of the attention interaction at the first level); performing attention interaction on the second feature (i.e., the visual feature of the second level) and the fourth feature (i.e., the output of the attention interaction at the first level) to obtain the fifth feature (i.e., the output of the attention interaction at the second level). If N is 2 in this example, then the level corresponding to the second feature is the lowest level among the two levels, and the fifth feature (i.e., the output of the attention interaction at the second level) can be used as the first interaction feature. If N is greater than 2 in this example, then the second feature in this example is not the lowest level among multiple levels, i.e., the Nth level. The fifth feature (i.e., the output of the attention interaction of the second level) can be input into the attention interaction of the next level to continue the interaction until the output of the attention interaction of the Nth level is obtained. Thus, the output of the attention interaction of the Nth level can be used as the first interaction feature.

[0157] In another example, the first feature can be a visual feature of the second level, and the second feature can be a visual feature of the third level. In this case, the level corresponding to the first feature is not the highest level among the N levels. Then the third feature is the output of the attention interaction of the level above the level corresponding to the first feature, that is, the first level. The process of cross-level attention interaction between the first text feature and the first visual feature in this case is the same as the process when the third feature is the first text feature, and will not be repeated here.

[0158] In addition, the first feature can also be a visual feature of other levels. In this case, we can refer to the case where the first feature is a visual feature of the second level, which will not be elaborated here.

[0159] The visual analysis method provided in this application uses first text features to interact with the highest-level visual features, and inputs the results of the interaction into the next level of attention interaction, progressing layer by layer, which helps to output more reliable features.

[0160] This application also provides a text encoder and a visual encoder, wherein the text encoder is used to extract text features, and the visual encoder can be used to extract visual features. In one possible implementation, step S101 may include: inputting first text data into the text encoder and outputting first text features; inputting first visual data into the visual encoder and outputting first visual features.

[0161] In this embodiment of the application, the extraction of the first text feature and the first visual feature can be achieved by using a trained visual analysis model. Figure 5 A schematic diagram of the structure of the visual analysis model provided in an embodiment of this application is shown. Figure 5 As shown, the visual analysis model can include a text encoder and a visual encoder. The video encoder is configured to extract features from the input visual data to obtain multiple levels of visual features, and the text encoder is configured to extract features from the input text data to obtain text features. Once the trained visual analysis model is obtained, the first text data can be input into the trained text encoder to output the first text features, and the first visual data can be input into the trained visual encoder to output the first visual features.

[0162] The visual analysis method provided in this application can determine more accurate text features and visual features through a well-trained text encoder and visual encoder, which is beneficial for subsequent interaction with text features and visual features and improves interaction capabilities.

[0163] The training process for the text encoder and the visual encoder will be explained in detail below.

[0164] In one possible implementation, the visual analysis method may further include training a text encoder and a visual encoder based on a training set. The training set may include multiple training sample pairs, each including second text data and second visual data.

[0165] In each training sample pair, the second text data can represent the semantic information of the second visual data. The second visual data can be a video and / or an image.

[0166] The process involves training the visual analysis model based on a training set, specifically training both the text encoder and the visual encoder until preset training conditions are met, at which point a trained visual analysis model is output. These preset training conditions can be related to the number of training iterations; when the number of training iterations reaches a preset training threshold, a trained visual analysis model can be output. Alternatively, the preset training conditions can be related to the model's loss function value; when the loss function value of the visual analysis model reaches a preset loss threshold or no longer changes, a trained visual analysis model can be output. The preset training threshold and loss threshold are set according to actual circumstances and are not intended to limit the scope of protection of this application.

[0167] The visual analysis method provided in this application is trained by pairwise related visual data and text data, which helps to obtain a text encoder and a visual encoder with higher accuracy, thereby helping to determine more accurate prediction results in practical applications.

[0168] like Figure 6 This diagram illustrates the first training process of the visual analysis model provided in an embodiment of this application. Figure 6 As shown, training a text encoder and a visual encoder based on a training set can include: for any training sample pair in the training set: inputting the second text data and the second visual data from the training sample pair into the text encoder and the visual encoder to be trained, respectively, and outputting second text features and second visual features, wherein the second visual features include visual features of multiple levels of the second visual data; performing cross-level attention interaction on the second text features and the second visual features to obtain second interaction features; determining the first loss corresponding to the training sample pair based on the second text features and the second interaction features; and adjusting the parameters of the text encoder and the visual encoder based on the first loss corresponding to each training pair.

[0169] The following operations can be performed on any training sample pair: The training sample pair can be input into the visual analysis model to be trained; that is, the second text data from the training sample pair can be input into the text encoder to be trained to obtain the second text features, and the second visual data from the training sample pair can be input into the visual encoder to be trained to obtain the second visual features. Cross-level attention interaction can be performed on the second text features and the second visual features to obtain the second interaction features. The cross-level attention interaction between text features and visual features has been detailed above and will not be repeated here. A first loss corresponding to the training sample pair can be determined based on the second text features and the second interaction features, and then the parameters of the visual analysis model can be updated using the first loss. In one possible implementation, the difference between the second text features and the second interaction features can be used to back-update the parameters of the text encoder and the visual encoder. Thus, the parameters of the text encoder and the visual encoder can be adjusted based on the first loss corresponding to each training pair in the training set until the preset training conditions are met, and the trained text encoder and visual encoder are output. The preset training conditions have been detailed above and will not be repeated here.

[0170] The visual analysis method provided in this application uses multiple training samples to train the text encoder and visual encoder, and uses the loss between the predicted result and the ideal result to back-update the parameters of the text encoder and visual encoder, which helps to obtain a more accurate text encoder and visual encoder, thereby obtaining a better visual analysis model.

[0171] like Figure 6As shown, the above training process may further include: for any training sample pair in the training set: performing feature contrast learning based on the second visual features to obtain the second loss corresponding to the training sample pair; and adjusting the parameters of the text encoder and the visual encoder based on the second loss corresponding to each training sample pair.

[0172] The following operations can be performed on any training sample pair: the second visual data in the training sample pair can be input into the visual encoder to be trained to obtain the second visual features. The second visual features include multiple levels of visual features. Feature contrast learning can be performed based on the multiple levels of visual features to obtain the second loss corresponding to the training sample pair, and then the parameters of the visual analysis model can be updated using the second loss.

[0173] Specifically, the parameters of the text encoder and the visual encoder can be adjusted based on the second loss corresponding to each training sample pair in the training set until the preset training conditions are met, outputting the trained text encoder and visual encoder. The preset training conditions have been detailed above and will not be repeated here.

[0174] The visual analysis method provided in this application uses a self-supervised learning mode of contrastive learning, which helps to avoid destroying visual information and affecting the attention interaction process during attention interaction. It also uses the results of contrastive learning to adjust the parameters of the visual analysis model, which is conducive to obtaining a more accurate visual analysis model. Thus, the visual analysis model can be used to perform better visual analysis.

[0175] Figure 7 This diagram illustrates the second training process of the visual analysis model provided in an embodiment of this application. Figure 7 As shown, the above training process may further include: performing feature comparison learning based on second visual features to obtain the second loss corresponding to the training sample pair, including: for any one of multiple levels: determining the global and local features of the level based on the visual features of the second visual data at the level; determining the second loss corresponding to the level based on the global and local features at the level; and determining the second loss corresponding to the training sample pair based on the second loss corresponding to each level.

[0176] Specifically, the following operations can be performed on any training sample pair: First, determine the corresponding second visual features based on the second visual data in the training sample pair. These second visual features include visual features at multiple levels. Second, determine the global and local features at each level based on the visual features at that level. Third, determine the second loss corresponding to each level based on the global and local features at that level. Finally, determine the total second loss for the training sample pair based on the second losses at each level, and then adjust the parameters of the text encoder and the visual encoder based on this total second loss.

[0177] The visual analysis method provided in this application not only utilizes global features but also fully mines local features across multiple levels, which enables better self-supervised learning. Furthermore, by employing a multi-level global-local comparative learning approach, visual representations become more robust, thereby further enhancing the interaction between different modal information.

[0178] In one possible implementation, determining the second loss corresponding to the level based on the global and local features of the level includes: determining the positive and negative features corresponding to the global features of the level from the local features of the level; and determining the second loss corresponding to the level based on the global features, positive features, and negative features.

[0179] Specifically, based on the second visual data in any training sample pair in the training set, a second visual feature can be obtained. This second visual feature includes visual features at multiple levels. For any given level, the following processing can be performed: the visual features at that level can be processed to obtain global and local features at that level; positive and negative features corresponding to the global features at that level can be determined from the local features; thus, the second loss corresponding to that level can be determined based on the global, positive, and negative features. This processing can be performed on each level of the second visual feature to determine the second loss for the training sample pair corresponding to the second visual feature. This processing can be performed on all training sample pairs in the training set to determine the total second loss for the entire training set.

[0180] In one example, taking any level of the second visual features as an example, this level has one global feature and multiple local features. Features that are the same as or similar to the global feature can be selected from these local features and designated as positive features corresponding to the global feature at this level. Alternatively, features can be randomly selected from these local features as positive features corresponding to the global feature at this level. Then, negative features can be randomly selected from the remaining local features as corresponding to the global feature at this level. The number of positive and negative features is not specifically limited.

[0181] In one example, the total second loss corresponding to the training set can be determined according to the following formula:

[0182]

[0183] In the formula, L represents the total second loss corresponding to the training set, which can include B sample sets, f ij As a global feature, Positive features Negative characteristics For global features f ij and positive characteristics Similarity between them For global features f ij and negative features The similarity between them, where ρ is the temperature hyperparameter.

[0184] Specifically, a trained visual analysis model can be output when the total second loss corresponding to the training set is less than a second threshold. The second threshold can be set based on experience or actual conditions, such that training the model using this second threshold yields a visual analysis model capable of solving the visual analysis problem; however, this does not limit the scope of protection of this application.

[0185] The visual analysis method provided in this application utilizes a multi-level global-local contrastive learning approach, which not only further enhances the robustness of visual modalities, thereby promoting cross-attention interaction between visual features and text features, but also supervises the training of the visual analysis model using contrastive learning, thus obtaining a high-precision visual analysis model, which is beneficial for subsequent visual analysis work.

[0186] The following section describes the process of acquiring the first text data and the first visual data, as well as the application process, based on the application scenario.

[0187] In one possible implementation, the visual analysis method may further include: upon receiving a visual data query request, obtaining first text data from the visual data query request and obtaining first visual data from the visual database to be queried; and if the first text data is semantic information of the first visual data, returning the first visual data.

[0188] Figure 8 A first flowchart of a visual analysis method provided according to an embodiment of this application is shown. Figure 8As shown, visual analysis methods can be used for video retrieval. When a visual data query request is received, first text data can be obtained from the request, and visual data can be obtained from the visual database to be queried. The first text data can be one or more keywords, or a complete or incomplete sentence. Visual data can include multiple videos. A trained visual analysis model can be used to perform visual analysis on any selected video and the first text data from multiple videos. The training process of the visual analysis model can be found above and will not be repeated here. If the first text data contains the semantic information of the selected video, the selected video is returned as the first visual data. One or more videos matching the first text data can be determined from multiple videos, and these videos can then be used as the result of the visual data query request. That is, the returned first visual data can include one or more videos. Taking a mobile phone as an example: the user enters "sunset" in the search box of the mobile app. The app calls the program related to the visual analysis method and displays videos related to "sunset," allowing the user to view the videos returned by the app.

[0189] The visual analysis method provided in this application can meet users' video retrieval needs and help users obtain more intuitive and three-dimensional video results compared to text and image results.

[0190] In one possible implementation, the visual analysis method may further include: upon receiving a visual data annotation request, obtaining first text data from a candidate tag library and first visual data from the visual data annotation request; and if the first text data is semantic information of the first visual data, returning the first text data.

[0191] Figure 9 A second flowchart of the visual analysis method provided according to an embodiment of this application is shown. Figure 9As shown, visual analysis methods can also be used for video annotation. When a visual data annotation request is received, first visual data can be obtained from the request, and tags (text data) can be obtained from a pre-defined candidate tag library. The first visual data can be one or more videos. The tag (text data) can be a word or a sentence. A trained visual analysis model can be used to perform visual analysis on any selected tag and first visual data from the text data. The training process of the visual analysis model can be found above and will not be repeated here. If the selected tag is the semantic information of the first visual data, the selected tag is returned as the first text data. One or more tags that match the first visual data can be selected from the candidate tag library, and these tags can then be used as the result of the visual data annotation request. That is, the returned first text data can include one or more tags.

[0192] The visual analysis method provided in this application can also tag massive amounts of unlabeled videos. The tag information can simultaneously contain fine-grained features and coarse-grained features, which is beneficial for applications such as video recommendation.

[0193] In one possible implementation, the visual analysis method may further include: upon receiving a visual data question-and-answer request, obtaining question information and first visual data from the visual data question-and-answer request, determining first text data based on the question information; and if the first text data is semantic information of the first visual data, returning the first text data.

[0194] Figure 10 A third flowchart of the visual analysis method provided according to an embodiment of this application is shown. Figure 10 As shown, visual analysis methods can also be used for video question answering. When a visual data question answering request is received, the first visual data and question information can be obtained from the request, and text data can be obtained from a pre-set text database. The first visual data can be one or more videos. The text data can be a word or a sentence. A trained visual analysis model can be used to perform visual analysis on the first visual data and text data, and the result of the visual data question answering request, i.e., the first text data, can be determined based on the question information. The training process of the visual analysis model can be found above and will not be repeated here.

[0195] The visual analysis method provided in this application embodiment can also provide corresponding answers to question information based on visual data, and can be widely applied in the field of machine vision, such as 3D digital human question answering.

[0196] As mentioned above, visual analysis methods can be applied to terminal devices, thereby facilitating users to perform operations such as video retrieval and video classification on these devices. It is understood that the training process involved in the embodiments of this application can be implemented by the terminal device, or by a server or the cloud; this application does not impose any limitations on this.

[0197] Figure 11 A schematic diagram of the structure of the visual analysis device provided in an embodiment of this application is shown. Figure 11 As shown, the visual analysis device 10 may include a feature generation unit 100, an attention interaction unit 200, and a semantic output unit 300. The feature generation unit 100 is configured to determine a first text feature based on first text data, and to determine a first visual feature based on first visual data. The first visual feature includes multiple levels of visual features from the first visual data, wherein the scales corresponding to the visual features at each level are different. The attention interaction unit 200 is configured to perform cross-level attention interaction on the first text feature and the first visual feature to obtain a first interaction feature. The semantic output unit 300 is configured to determine the first text data as semantic information of the first visual data if the similarity between the first text feature and the first interaction feature is greater than a first threshold.

[0198] Compared to using visual features and text features at a single scale for interaction, the visual analysis device 10 provided in this application uses visual features and text features at multiple scales for interaction. The visual features can simultaneously cover coarse-grained features and fine-grained features, which helps to significantly improve the interaction capabilities between different modalities, thereby enabling accurate extraction of semantic information from visual data. In addition, using multiple sets of attention interactions can reduce the use of graphics processing unit (GPU) memory.

[0199] In one possible implementation, cross-level attentional interactions include attentional interactions between two adjacent levels, wherein the input of the attentional interaction of the next level is the output of the attentional interaction of the previous level.

[0200] The visual analysis device provided in this application embodiment sets up cross-level attention interaction, inputting the output of the attention interaction of the previous level into the attention interaction of the next level. At the same time, it uses deep semantic features and shallow fine-grained features for text interaction, which helps to improve the output accuracy of attention interaction, so that the first interaction feature contains the semantic features of the first visual data.

[0201] In one possible implementation, the visual features of the first visual data at multiple levels include at least a first feature and a second feature, wherein the levels corresponding to the first feature and the second feature are adjacent and the level corresponding to the first feature is higher than the level corresponding to the second feature, and the attention interaction unit is further configured as follows:

[0202] Attention interaction is performed on the first feature and the third feature to obtain the fourth feature. Wherein, if the level corresponding to the first feature is the highest level among multiple levels, the third feature is the first text feature. If the level corresponding to the first feature is not the highest level among multiple levels, the third feature is the output of the level above the level corresponding to the first feature.

[0203] The fifth feature is obtained by performing attention interaction between the second and fourth features;

[0204] If the level corresponding to the second feature is the lowest level among multiple levels, the fifth feature is used as the first interactive feature.

[0205] If the level corresponding to the second feature is not the lowest level among multiple levels, the fifth feature is input into the next level of the level corresponding to the second feature for attention interaction until the first interaction feature is obtained.

[0206] The visual analysis device provided in this application uses first text features to interact with the highest-level visual features, and inputs the results of the interaction into the next level of attention interaction, progressing layer by layer, which helps to output more reliable features.

[0207] In one possible implementation, the feature generation unit is also configured as follows:

[0208] The first text data is input into the text encoder, and the first text feature is output.

[0209] The first visual data is input into the visual encoder, and the first visual features are output.

[0210] The visual analysis device provided in this application embodiment can determine more accurate text features and visual features through a trained text encoder and visual encoder, which is beneficial for subsequent interaction with text features and visual features and improves interaction capabilities.

[0211] In one possible implementation, the visual analysis device further includes:

[0212] The training unit is configured to train a text encoder and a visual encoder based on a training set, the training set including multiple training sample pairs, each training sample pair including second text data and second visual data.

[0213] The visual analysis device provided in this application embodiment is trained by pairwise related visual data and text data, which is beneficial to obtaining a text encoder and a visual encoder with higher accuracy, thereby helping to determine more accurate prediction results in practical applications.

[0214] In one possible implementation, the training unit is also configured as follows:

[0215] For any training sample pair in the training set:

[0216] The second text data and the second visual data in the training sample pair are respectively input into the text encoder and the visual encoder to be trained, and the second text features and the second visual features are output. The second visual features include multiple levels of visual features of the second visual data.

[0217] Cross-level attention interaction is performed on the second text feature and the second visual feature to obtain the second interactive feature;

[0218] Based on the second text features and the second interaction features, determine the first loss corresponding to the training sample pair;

[0219] Based on the first loss corresponding to each training pair, adjust the parameters of the text encoder and the visual encoder.

[0220] The visual analysis device provided in this application uses multiple training samples to train the text encoder and the visual encoder, and uses the loss between the predicted result and the ideal result to back-update the parameters of the text encoder and the visual encoder, which helps to obtain a more accurate text encoder and visual encoder, thereby obtaining a better visual analysis model.

[0221] In one possible implementation, the visual analysis device further includes:

[0222] The contrast unit is configured to perform feature contrast learning based on the second visual features for any training sample pair in the training set, and obtain the second loss corresponding to the training sample pair.

[0223] The adjustment unit is configured to adjust the parameters of the text encoder and the visual encoder based on the second loss corresponding to each training sample pair.

[0224] The visual analysis device provided in this application uses a self-supervised learning mode of contrastive learning, which helps to avoid destroying visual information and affecting the attention interaction process during attention interaction. It also uses the results of contrastive learning to adjust the parameters of the visual analysis model, which is conducive to obtaining a more accurate visual analysis model. Thus, the visual analysis model can be used to perform better visual analysis.

[0225] In one possible implementation, the comparison unit is also configured as follows:

[0226] For any one of the multiple levels:

[0227] Based on the visual features of the second visual data at the level, determine the global and local features at the level;

[0228] Based on the global and local features under the hierarchy, determine the second loss corresponding to the hierarchy;

[0229] Based on the second loss corresponding to each level, determine the second loss corresponding to the training sample pair.

[0230] The visual analysis device provided in this application not only utilizes global features but also fully mines local features across multiple levels, which enables better self-supervised learning. Furthermore, by employing a multi-level global-local comparative learning approach, visual representations become more robust, thereby further enhancing the interaction between different modal information.

[0231] In one possible implementation, the second loss corresponding to the level is determined based on the global and local features at that level, including:

[0232] From the local features at the hierarchical level, determine the positive and negative features corresponding to the global features at the hierarchical level;

[0233] Based on global features, positive features, and negative features, determine the second loss corresponding to each level.

[0234] The visual analysis device provided in this application utilizes a multi-level global-local contrastive learning approach, which not only further enhances the robustness of visual modalities, thereby promoting cross-attention interaction between subsequent visual features and text features, but also supervises the training of the visual analysis model using contrastive learning, thus obtaining a high-precision visual analysis model, which is beneficial for subsequent visual analysis work.

[0235] In one possible implementation, the visual analysis device further includes:

[0236] The first acquisition unit is configured to, upon receiving a visual data query request, acquire first text data from the visual data query request and acquire first visual data from the visual database to be queried.

[0237] The first return unit is configured to return the first visual data when the first text data is semantic information of the first visual data;

[0238] And / or,

[0239] The second acquisition unit is configured to, upon receiving a visual data annotation request, acquire first text data from a candidate label library and first visual data from a visual data annotation request.

[0240] The second return unit is configured to return the first text data when the first text data is semantic information of the first visual data;

[0241] And / or,

[0242] The third acquisition unit is configured to, upon receiving a visual data question-and-answer request, acquire question information and first visual data from the visual data question-and-answer request, and determine first text data based on the question information;

[0243] The third return unit is configured to return the first text data if the first text data is semantic information of the first visual data.

[0244] The visual analysis device provided in this application not only meets users' video retrieval needs and helps users obtain more intuitive and three-dimensional video results compared to text and image results, but also can tag massive amounts of unlabeled videos. The tag information can simultaneously contain fine-grained and coarse-grained features, which is beneficial for applications such as video recommendation. Furthermore, it can provide corresponding answers to question information based on visual data and can be widely used in the field of machine vision, such as 3D digital human question answering.

[0245] The idea behind this visual analysis device embodiment is the same as the working process of the visual analysis method in the above embodiments. All contents of the above visual analysis method embodiments are incorporated into this visual analysis device embodiment by means of full reference, and will not be repeated.

[0246] An embodiment of this application provides a terminal device, including: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described visual analysis method when executing the instructions.

[0247] The concept of this terminal device embodiment is the same as the working process of the visual analysis method in the above embodiments. All the contents of the above visual analysis method embodiments are incorporated into this terminal device embodiment by means of full reference, and will not be repeated.

[0248] Figure 12 A schematic diagram of the structure of a terminal device provided in an embodiment of this application is shown. Figure 12 As shown, the terminal device may include at least one processor 301, a memory 302, an input / output device 303, and a bus 304. The following is in conjunction with... Figure 12A detailed introduction to each component of the terminal device:

[0249] Processor 301 is the control center of the terminal device. It can be a single processor or a collective term for multiple processing elements. For example, processor 301 can be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this disclosure, such as one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs).

[0250] The processor 301 can perform various functions of the terminal device by running or executing software programs stored in the memory 302 and calling data stored in the memory 302.

[0251] In a specific implementation, as one example, processor 301 may include one or more CPUs, such as CPU 0 and CPU 1 shown in the figure.

[0252] In a specific implementation, as one example, the terminal device may include multiple processors (not shown). Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0253] Memory 302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 302 may exist independently and be connected to processor 301 via bus 304. Memory 302 may also be integrated with processor 301.

[0254] Input / output device 303 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), and Wireless Local Area Networks (WLAN). Input / output device 303 may include all or part of a baseband processor, and may optionally include a Radio Frequency (RF) processor. The RF processor is used to transmit and receive RF signals, while the baseband processor is used to process baseband signals converted from RF signals or baseband signals that are about to be converted to RF signals.

[0255] In a specific implementation, as one example, the input / output device 303 may include a transmitter and a receiver. The transmitter is used to send signals to other devices or communication networks, and the receiver is used to receive signals sent by other devices or communication networks. The transmitter and receiver may exist independently or be integrated together.

[0256] Bus 304 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0257] Figure 12 The device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0258] Embodiments of this application provide a non-volatile computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the above-described visual analysis method.

[0259] The idea behind this non-volatile computer-readable storage medium embodiment is the same as the working process of the visual analysis method in the above embodiments. All contents of the above-described visual analysis method embodiments are incorporated into this non-volatile computer-readable storage medium embodiment by means of full reference, and will not be repeated here.

[0260] Embodiments of this application provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described visual analysis method.

[0261] The idea behind this computer program product embodiment is the same as the working process of the visual analysis method in the above embodiments. All contents of the above visual analysis method embodiments are incorporated into this computer program product embodiment by means of full reference, and will not be repeated.

[0262] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing.

[0263] The computer-readable program instructions or code described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0264] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this application.

[0265] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0266] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0267] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0268] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0269] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0270] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0271] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A visual analysis method, characterized by, The method comprises: determining a first text feature according to first text data, and determining a first visual feature according to first visual data, the first visual feature comprising visual features of multiple levels of the first visual data, wherein corresponding scales of visual features of respective levels are different; performing cross-level attention interaction on the first text feature and the first visual feature to obtain a first interaction feature; in a case where a similarity between the first text feature and the first interaction feature is greater than a first threshold, determining the first text data as semantic information of the first visual data; the multiple levels of visual features of the first visual data at least comprise a first feature and a second feature, the first feature and the second feature correspond to adjacent levels and the first feature corresponds to a higher level than the second feature, and the performing cross-level attention interaction on the first text feature and the first visual feature comprises: performing attention interaction on the first feature and a third feature to obtain a fourth feature, wherein, in a case where the level corresponding to the first feature is a highest level of the multiple levels, the third feature is the first text feature, and in a case where the level corresponding to the first feature is not the highest level of the multiple levels, the third feature is an output of a level next to the level corresponding to the first feature; performing attention interaction on the second feature and the fourth feature to obtain a fifth feature; in a case where the level corresponding to the second feature is a lowest level of the multiple levels, taking the fifth feature as the first interaction feature; in a case where the level corresponding to the second feature is not the lowest level of the multiple levels, inputting the fifth feature into a level next to the level corresponding to the second feature to perform attention interaction until the first interaction feature is obtained.

2. The method of claim 1, wherein, The cross-level attention interaction comprises attention interaction of adjacent two levels, wherein, in the attention interaction of the adjacent two levels, an input of attention interaction of a next level is an output of attention interaction of a previous level.

3. The method of claim 1, wherein, The determining a first text feature according to first text data, and determining a first visual feature according to first visual data comprises: inputting the first text data into a text encoder to output the first text feature; inputting the first visual data into a visual encoder to output the first visual feature.

4. The method of claim 3, wherein, The method further comprises: training the text encoder and the visual encoder based on a training set, the training set comprising multiple training sample pairs, each training sample pair comprising second text data and second visual data.

5. The method of claim 4, wherein, The training the text encoder and the visual encoder based on the training set comprises: for any one training sample pair in the training set: inputting the second text data and the second visual data in the training sample pair into a text encoder and a visual encoder to be trained respectively to output a second text feature and a second visual feature, the second visual feature comprising visual features of multiple levels of the second visual data; perform cross-level attention interaction on the second text feature and the second visual feature to obtain a second interaction feature; determine a first loss corresponding to the training sample pair based on the second text feature and the second interaction feature; adjust parameters of the text encoder and parameters of the visual encoder based on the first loss corresponding to each training pair.

6. The method of claim 5, wherein, The method further includes: perform feature contrast learning based on the second visual feature to obtain a second loss corresponding to the training sample pair; adjust the parameters of the text encoder and the parameters of the visual encoder based on the second loss corresponding to each training sample pair.

7. The method of claim 6, wherein, The feature contrast learning based on the second visual feature to obtain the second loss corresponding to the training sample pair includes: for any one of the plurality of levels: determine global features and local features under the level according to the visual features of the second visual data at the level; determine a second loss corresponding to the level according to the global features and the local features under the level; determine the second loss corresponding to the training sample pair according to the second loss corresponding to each level.

8. The method of claim 7, wherein, The determination of the second loss corresponding to the level according to the global features and the local features under the level includes: determine positive features and negative features corresponding to the global features under the level from the local features under the level; determine the second loss corresponding to the level based on the global features, the positive features and the negative features.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: in the case of receiving a visual data query request, obtaining the first text data from the visual data query request and obtaining the first visual data from a visual database to be queried; in the case that the first text data is semantic information of the first visual data, returning the first visual data; and / or, in the case of receiving a visual data annotation request, obtaining the first text data from a candidate label library and obtaining the first visual data from the visual data annotation request; in the case that the first text data is semantic information of the first visual data, returning the first text data; and / or, in the case of receiving a visual data question and answer request, obtaining question information and the first visual data from the visual data question and answer request, and determining the first text data based on the question information; in the case that the first text data is semantic information of the first visual data, returning the first text data.

10. A visual analysis apparatus, characterized by comprise: a feature generation unit configured to determine a first text feature according to first text data, and determine a first visual feature according to first visual data, the first visual feature comprising visual features of a plurality of levels of the first visual data, wherein the corresponding scales of the visual features of each level are different; an attention interaction unit configured to perform cross-level attention interaction on the first text feature and the first visual feature to obtain a first interaction feature; a semantic output unit configured to determine the first text data as semantic information of the first visual data in a case where the similarity between the first text feature and the first interaction feature is greater than a first threshold value; the visual features of the multiple levels of the first visual data at least include a first feature and a second feature, the first feature and the second feature correspond to adjacent levels and the first feature corresponds to a higher level than the second feature, and the attention interaction unit is further configured to: perform attention interaction on the first feature and a third feature to obtain a fourth feature, wherein, in a case where the level corresponding to the first feature is the highest level of the multiple levels, the third feature is the first text feature, and in a case where the level corresponding to the first feature is not the highest level of the multiple levels, the third feature is the output of the level one level above the level corresponding to the first feature; perform attention interaction on the second feature and the fourth feature to obtain a fifth feature; in a case where the level corresponding to the second feature is the lowest level of the multiple levels, the fifth feature is taken as the first interaction feature; in a case where the level corresponding to the second feature is not the lowest level of the multiple levels, the fifth feature is input into the level one level below the level corresponding to the second feature to perform attention interaction until the first interaction feature is obtained.

11. The apparatus of claim 10, wherein, The cross-level attention interaction includes attention interaction of two adjacent levels, wherein, in the attention interaction of the two adjacent levels, the input of the attention interaction of the next level is the output of the attention interaction of the previous level.

12. The apparatus of claim 10, wherein, The feature generation unit is further configured to: input the first text data into a text encoder to output the first text feature; input the first visual data into a visual encoder to output the first visual feature.

13. The apparatus of claim 12, wherein, The apparatus further includes: a training unit configured to train the text encoder and the visual encoder based on a training set, the training set including multiple training sample pairs, each training sample pair including second text data and second visual data.

14. The apparatus of claim 13, wherein, The training unit is further configured to: for any one training sample pair in the training set: input the second text data and the second visual data in the training sample pair into the text encoder and the visual encoder to be trained respectively to output a second text feature and a second visual feature, the second visual feature including visual features of multiple levels of the second visual data; perform cross-level attention interaction on the second text feature and the second visual feature to obtain a second interaction feature; determine a first loss corresponding to the training sample pair based on the second text feature and the second interaction feature; adjust parameters of the text encoder and parameters of the visual encoder based on the first loss corresponding to each training pair.

15. The apparatus of claim 14, wherein, The apparatus further includes: a comparison unit configured to perform feature comparison learning based on the second visual features for any one training sample pair in the training set to obtain a second loss corresponding to the training sample pair; an adjustment unit configured to adjust parameters of the text encoder and parameters of the visual encoder based on the second loss corresponding to each training sample pair.

16. The apparatus of claim 15, wherein, The comparison unit is further configured to: for any one level in the plurality of levels: determine global features and local features under the level according to visual features of the level based on the second visual data; determine a second loss corresponding to the level according to the global features and the local features under the level; determine the second loss corresponding to the training sample pair according to the second loss corresponding to each level.

17. The apparatus of claim 16, wherein, The determination of the second loss corresponding to the level according to the global features and the local features under the level comprises: determining positive features and negative features corresponding to the global features under the level from the local features under the level; determining the second loss corresponding to the level based on the global features, the positive features and the negative features.

18. The apparatus of any one of claims 10 to 17, wherein, The apparatus further comprises: a first acquisition unit configured to acquire the first text data from a visual data query request and acquire the first visual data from a visual database to be queried when the visual data query request is received; a first return unit configured to return the first visual data when the first text data is semantic information of the first visual data; and / or, a second acquisition unit configured to acquire the first text data from a candidate label library and acquire the first visual data from a visual data annotation request when the visual data annotation request is received; a second return unit configured to return the first text data when the first text data is semantic information of the first visual data; and / or, a third acquisition unit configured to acquire question information and the first visual data from a visual data question and answer request and determine the first text data based on the question information when the visual data question and answer request is received; a third return unit configured to return the first text data when the first text data is semantic information of the first visual data.

19. A terminal device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method of any one of claims 1 to 9 when executing the instructions.

20. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1 to 9.

21. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in an electronic device, a processor in the electronic device performs the method of any one of claims 1 to 9.

Images