Method and device for processing video data

Abstract

The invention relates to a video data processing method, a neural network training method, a video data processing device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, fine-grained fusion of the pixel-level visual features and the text features on a multi-semantic level is realized. In one optional aspect, the embodiments of the present disclosure fuse textual features with visual features in a hierarchical and fine-grained manner in an early stage of visual feature and textual feature processing. According to the embodiment of the invention, the text data can be fully applied, and the performance of the RMOT task is greatly improved. In another alternative aspect, according to the embodiment of the invention, the text is utilized to guide the decoder, and the prior semantic information is provided for the query in the decoding process, so that the decoder can more fully utilize the semantic information in the text data, and the embedding expression capability of the target to be identified is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence services, and more specifically to a method for processing video data, a method for training a neural network, a device for processing video data, an electronic device, and a computer-readable storage medium. Background Art

[0002] Referenced Multi-Object Tracking (RMOT) is a computer vision technique that identifies and tracks multiple specific objects through natural language descriptions. Unlike traditional multi-object tracking (MOT) that mainly relies on the visual features of the objects (such as appearance, position, motion trajectory, etc.), the RMOT task requires understanding natural language and matching textual information with visual objects. This capability makes RMOT have great potential in handling personalized tracking needs in complex scenarios, especially in application areas such as human-computer interaction, intelligent monitoring and autonomous driving.

[0003] At present, most of the research in the field of RMOT is devoted to combining text features with the multi-target tracking framework, and realizing multimodal collaboration by combining text features and visual features. However, existing methods often use simple feature fusion techniques, such as directly splicing text features and visual features together. This rough processing method may weaken the representation of semantic information in text features after fusion, resulting in the failure to fully utilize text features, thereby limiting the performance of RMOT in complex scenarios. Summary of the invention

[0004] The embodiments of the present disclosure provide a method for processing video data, a method for training a neural network, an apparatus for processing video data, an electronic device, and a computer-readable storage medium.

[0005] An embodiment of the present disclosure provides a method for processing video data, the method comprising: acquiring video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data comprises a plurality of video frames arranged in time sequence; determining a first text feature and a second text feature corresponding to the text data based on the text data, wherein the first text feature and the second text feature correspond to different semantic levels; determining a video feature corresponding to the video frame based on the video frame in the video data, the video feature comprising a visual feature corresponding to each pixel in the video frame; fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; fusing the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; determining an identification feature for identifying the target to be identified in the video data based on the video feature, the first fused feature and the second fused feature; and identifying the target to be identified in the video frame based on the identification feature.

[0006] The embodiment of the present disclosure provides a method for training a neural network, wherein the neural network model includes a multimodal fusion module and a target recognition module, wherein the method includes: obtaining a video sample and a text sample corresponding to the video sample, wherein the text sample indicates a target to be recognized in the video sample, and the video sample includes a plurality of video frames arranged in time sequence; based on the text sample, using the multimodal fusion module, determining a first text feature and a second text feature corresponding to the text sample, wherein the semantic levels corresponding to the first text feature and the second text feature are different; based on the video frame in the video sample, determining a video feature corresponding to the video frame, wherein the video feature includes a visual feature corresponding to each pixel in the video frame; using The multimodal fusion module fuses the visual features corresponding to each pixel in the video frame with the first text feature to determine the first fusion feature; uses the multimodal fusion module to fuse the visual features corresponding to each pixel in the video frame with the second text feature to determine the second fusion feature; uses the multimodal fusion module to determine the identification features for identifying the target to be identified in the video sample based on the video features, the first fusion features and the second fusion features; based on the identification features, uses the target identification module to determine the predicted value of the target to be identified in the video frame; and based on the difference between the predicted value and the true value of the target to be identified, trains at least one of the multimodal fusion module and the target identification module.

[0007] The embodiment of the present disclosure provides a device for processing video data, the device comprising: an acquisition module, configured to: acquire video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data comprises a plurality of video frames arranged in time sequence; a video frame feature extraction module, configured to: determine, based on the video frames in the video data, the video features corresponding to the video frames, wherein the video features comprise visual features corresponding to each pixel in the video frames; a multimodal fusion module, configured to: determine, based on the text data, a first text feature and a second text feature corresponding to the text data, wherein the first text feature and the second text feature correspond to different semantic levels; fuse the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; fuse the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; determine, based on the video feature, the first fused feature and the second fused feature, an identification feature for identifying the target to be identified in the video data; and a target identification module, configured to: identify the target to be identified in the video frame based on the identification feature.

[0008] An embodiment of the present disclosure discloses an electronic device, including: one or more processors; and one or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above method is performed.

[0009] An embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above method is implemented.

[0010] According to another aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various aspects or various optional implementations of the above-mentioned various aspects.

[0011] The disclosed embodiments achieve fine-grained fusion of pixel-level visual features and text features at multiple semantic levels. In an optional aspect, the disclosed embodiments fuse text features with visual features in a hierarchical and fine-grained manner in the early stages of visual feature and text feature processing. As a result, the disclosed embodiments can fully utilize text data and significantly improve the performance of the RMOT task.

[0012] In another optional aspect, the disclosed embodiment utilizes a text-guided decoder (TGD) to further integrate prior semantic information in the decoding process, so that the decoder can more fully utilize the semantic information in the text data and improve the embedding performance of the target to be identified.

[0013] In another optional aspect, the disclosed embodiment further introduces a text-guided prediction method to enhance the prediction capability of the RMOT task by utilizing text information.

[0014] In another optional aspect, further, during the training phase, the disclosed embodiment adopts a backtracking training strategy to maximize the utilization of valuable annotated data, thereby improving the overall performance of the RMOT task. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following briefly introduces the drawings required for describing the embodiment. The drawings described below are only exemplary embodiments of the present disclosure.

[0016] Figure 1 is an example schematic diagram showing a scenario according to an embodiment of the present disclosure.

[0017] Figure 2 A schematic diagram of an example RMOT task according to an embodiment of the present disclosure is shown.

[0018] Figure 3 A flowchart of a method for processing video data according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A schematic diagram of a multimodal fusion module according to an embodiment of the present disclosure is shown.

[0020] Figure 5 The figure shows an overall architecture diagram of a neural network model for performing RMOT tasks according to an embodiment of the present disclosure.

[0021] Figure 6 A schematic diagram of a decoder of an improved neural network model according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A schematic diagram of a prediction head of an improved neural network model according to an embodiment of the present disclosure is shown.

[0023] Figure 8 A schematic diagram of an improved neural network model training method according to an embodiment of the present disclosure is shown.

[0024] Fig. 9 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0025] Fig.10 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure.

[0026] Fig.11 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0028] In this specification and the accompanying drawings, operations and elements having substantially the same or similar features are denoted by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the descriptions, and cannot be understood as indicating or implying relative importance or ranking.

[0029] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0030] Optionally, the models that can be used in the embodiments of the present disclosure below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, the neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating outputs in the output layer. The nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layer, activation layer, pooling layer, convolution layer, etc. The input and output of the neural network model can take various forms, and the present disclosure is not limited to this.

[0031] The solutions provided in the embodiments of the present disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically described through the following embodiments.

[0032] First, refer to Figure 1 The application scenarios of the method for processing video data and the corresponding device according to the embodiments of the present disclosure are described. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically shown.

[0033] The neural network model of the embodiment of the present disclosure can be integrated into various electronic devices, for example, Figure 1 The server 110 and any electronic device in the multiple terminals 120. For example, the neural network model can be integrated in the terminal 120. The terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC, Personal Computer), a smart speaker or a smart watch, etc., but is not limited to this. For another example, the neural network model can also be integrated in the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this disclosure.

[0034] It is understandable that the device for reasoning using the neural network model of the embodiment of the present disclosure can be a terminal, a server, or a system composed of a terminal and a server. The method for processing video data of the embodiment of the present disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0035] The artificial intelligence model provided by the embodiment of the present disclosure may also involve artificial intelligence cloud services in the field of cloud technology. It is worth noting that the terminal 110 and the server 120 according to the embodiment of the present disclosure comply with the data protection principle, respect the user's data rights, and protect the user's data security and privacy. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will clearly inform the user of the purpose, method and scope of collecting, using, storing, transmitting and deleting the user's data, and obtain the user's consent. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will take reasonable technical and management measures to prevent the user's data from being leaked, tampered with, damaged or lost. According to the embodiment of the present disclosure, the terminal 110 and the server 120 provider will regularly review and update the user's data and delete expired or useless data in a timely manner. In addition, the cloud service provider using the embodiment of the present disclosure respects the user's rights to access, correct, delete, withdraw consent, complain and claim data, and provides convenient channels and procedures so that users can effectively exercise these rights.

[0036] Furthermore, the process of using artificial intelligence technology to analyze data in the terminal 120 or the server 110 is based on the principles of legality, rationality and transparency. The data collected and processed by the artificial intelligence model according to the embodiment of the present disclosure is relevant, necessary and appropriate for the prediction purpose and does not contain any personal identity information or sensitive information. The neural network model according to the embodiment of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent the data from being accessed, used or leaked without authorization.

[0037] The artificial intelligence-based neural network model according to the embodiments of the present disclosure will comply with relevant data protection regulations and ethical principles. The neural network model is trained based on a large amount of anonymized and de-identified data and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also been rigorously tested and evaluated to ensure that its output results are accurate and reliable and will not cause any misleading or discrimination. The artificial intelligence model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. In addition, the neural network model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.

[0038] Referenced Multi-Object Tracking (RMOT) is an emerging technology that attempts to make up for the deficiencies in flexibility and generalization of traditional multi-object tracking (MOT) tasks. The core task of RMOT is to find and track specific objects that match a given text description in each frame and assign a unique identification to these objects. Although RMOT is still in its early stages of development, it has attracted widespread attention in the industry due to its great potential in multiple fields such as autonomous driving, human-computer interaction, and animal research.

[0039] At present, the research on RMOT mainly focuses on how to integrate text features into the framework of multi-target tracking, and realize multimodal cooperation by combining text information and visual information. However, most existing methods use simple feature fusion techniques, such as directly splicing text features and visual features together. This rough processing method may ignore the importance of semantic information in text features, resulting in insufficient utilization of semantic information, which in turn affects the performance of RMOT in complex environments. In addition, the analysis and application of RMOT datasets are not sufficient, which may lead to the inability to fully utilize valuable text data during training. Therefore, the robustness and effect of RMOT tasks are often difficult to reach the expected level.

[0040] The disclosed embodiments achieve fine-grained fusion of pixel-level visual features and text features at multiple semantic levels. In an optional aspect, the disclosed embodiments fuse text features with visual features in a hierarchical and fine-grained manner in the early stage of feature processing. In another optional aspect, the disclosed embodiments utilize a text-guided decoder (TGD) to provide prior semantic information for queries during the decoding process. In another optional aspect, the disclosed embodiments further introduce a text-guided prediction head (TGPH) to enhance the performance of the prediction module by utilizing text information. In another optional aspect, further, in the training phase, the disclosed embodiments adopt a backtracking training strategy to maximize the utilization of valuable annotated data, thereby improving the overall performance of the RMOT task.

[0041] The embodiment of the present disclosure proposes an improved neural network model. The improved neural network model includes at least a multimodal fusion module and a target recognition module. The multimodal fusion module is used to achieve the fusion of text features and visual features in a hierarchical and fine-grained manner. The target recognition module implements the RMOT task based on the fused features. Of course, the present disclosure is not limited to this.

[0042] For the improved neural network model, the embodiment of the present disclosure provides a method for processing video data to perform reasoning based on such an improved neural network model. The method includes: obtaining video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data includes a plurality of video frames arranged in time sequence; based on the text data, determining a first text feature and a second text feature corresponding to the text data, wherein the semantic levels corresponding to the first text feature and the second text feature are different; based on the video frame in the video data, determining the video feature corresponding to the video frame, wherein the video feature includes a visual feature corresponding to each pixel in the video frame; fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; fusing the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; based on the video feature, the first fused feature and the second fused feature, determining a recognition feature for identifying the target to be identified in the video data; and based on the recognition feature, identifying the target to be identified in the video frame.

[0043] In the process of reasoning using the improved neural network, the disclosed embodiment achieves fine-grained fusion of pixel-level visual features and text features at multiple semantic levels, thereby making full use of text data and significantly improving the performance of the RMOT task.

[0044] Similarly, for the improved neural network model, the embodiment of the present disclosure also provides a method for training a neural network to train the improved neural network model. The neural network model includes a multimodal fusion module and a target recognition module, wherein the method includes: obtaining a video sample and a text sample corresponding to the video sample, wherein the text sample indicates a target to be recognized in the video sample, and the video sample includes a plurality of video frames arranged in time sequence; based on the text sample, using the multimodal fusion module, determining a first text feature and a second text feature corresponding to the text sample, wherein the semantic levels corresponding to the first text feature and the second text feature are different; based on the video frame in the video sample, determining the video feature corresponding to the video frame, wherein the video feature includes the visual feature corresponding to each pixel in the video frame; using the multimodal fusion module, The visual feature corresponding to each pixel in the video frame is fused with the first text feature to determine a first fused feature; the visual feature corresponding to each pixel in the video frame is fused with the second text feature using the multimodal fusion module to determine a second fused feature; the identification feature for identifying the target to be identified in the video sample is determined using the multimodal fusion module based on the video feature, the first fused feature and the second fused feature; based on the identification feature, the predicted value of the target to be identified in the video frame is determined using the target identification module; and based on the difference between the predicted value and the true value of the target to be identified, at least one of the multimodal fusion module and the target identification module is trained.

[0045] During the training phase, the disclosed embodiment adopts a backtracking training strategy to maximize the utilization of valuable annotated data, thereby improving the overall performance of the RMOT task.

[0046] Refer to the following Figures 2 to 11 A method for processing video data, a method for training a neural network, and an apparatus are described according to an embodiment of the present disclosure.

[0047] Figure 2 A schematic diagram of an example RMOT task according to an embodiment of the present disclosure is shown.

[0048] The RMOT task can be widely used in a variety of scenarios, such as real-time identification and tracking of specific vehicles in autonomous driving systems to improve road safety, rapid location of suspects or missing persons based on descriptions in intelligent surveillance systems to enhance public safety, and tracking of specific animal groups in wildlife research for ecological research and protection.

[0049] by Figure 2Taking the traffic monitoring system shown in the figure as an example, the RMOT task can be used to analyze street traffic video streams in real time and pay special attention to specific types of vehicles. For example, the task input received by the traffic monitoring system includes a video recording a busy street and a specific reference description: "a black car in motion". The task output of the system is to use boxes to clearly identify all black vehicles that meet the reference description on each frame of the video. These boxes will be dynamically adjusted as the vehicles move, ensuring that the traffic monitoring system can accurately track each black vehicle even in a complex traffic environment. In this way, the RMOT task improves the overall management capabilities of the traffic monitoring system to identify and monitor specific types of vehicles, assisting public management personnel in safety monitoring and data analysis.

[0050] Taking the application of RMOT tasks to tracking specific animal groups as an example, researchers may give a reference description, such as "a group of antelopes running on the grassland", and the RMOT system needs to identify and track all antelopes that meet this description in field surveillance videos. The system will mark all matching antelopes in each frame of the video and update the mark position in real time as the antelopes move, so as to study animal behavior patterns, population dynamics, and ecological protection planning. In this way, RMOT technology can not only improve the efficiency and accuracy of ecological research, but also help conservationists better study wildlife groups.

[0051] Figure 3 A flowchart of a method 30 for processing video data according to an embodiment of the present disclosure is shown.

[0052] Method 30 can be performed on a terminal device or a server (such as Figure 1 The method 30 includes the following operations S301 to S307. Of course, the method 30 may also include more operations or fewer operations, and the present disclosure is not limited thereto.

[0053] In operation S301, video data and text data corresponding to the video data are acquired, wherein the text data indicates a target to be recognized in the video data, and the video data includes a plurality of video frames arranged in time sequence.

[0054] Optionally, video data refers to a sequence consisting of a series of continuous images (i.e., video frames). In some systems that apply RMOT technology, video frames are arranged in chronological order to form a video stream. Specifically, each video frame can be regarded as a frame in the video data. For example, in a traffic monitoring system, a video frame may contain elements such as cars, pedestrians, and traffic signals on a busy street, and each frame is a static image of the street at different time points.

[0055] Optionally, text data provides descriptive information about the video content, which indicates specific targets of interest in each video frame. This text data can be simple keywords such as "black vehicle" or more complex descriptions such as "buses traveling during rush hour" or even a paragraph. For example, in a traffic monitoring system, text data may contain descriptions of specific vehicle colors, models, or behaviors to assist the traffic monitoring system in identifying and tracking specific targets in the video.

[0056] Optionally, the targets to be identified refer to those objects that need to be paid special attention to and identified in the video frame according to the description of text data, that is, "specific targets of interest". These targets can be people, vehicles, animals or any other entities that can be observed in the video scene. When defining the targets to be identified using text data, not only the appearance features of the targets to be identified can be limited, but also the behavior patterns, position changes, etc. of the targets to be identified may be limited. For example, in an autonomous driving system, the targets to be identified may be pedestrians or obstacles in front, while in ecological research, the targets to be identified may be a group of animals of a specific type. The present disclosure is not limited to this.

[0057] In operation S302, based on the text data, a first text feature and a second text feature corresponding to the text data are determined, wherein the first text feature and the second text feature correspond to different semantic levels.

[0058] Optionally, semantic levels refer to different levels in text analysis, divided according to the degree of abstraction and scope of semantics. These levels range from specific lexical meanings to overall text semantics. The first text feature is, for example, a word-level text feature (hereinafter also referred to as a word feature) and the second text feature (hereinafter also referred to as a sentence feature). Of course, the present disclosure is not limited to this.

[0059] For example, in the case where the text data is "a black car in the process of moving", the first text feature (word feature) will include word units such as "black", "of", "moving", "process", "in", "of", "car", etc., and each word unit separately represents the semantic information of the word. These word features together constitute the basic semantic unit of the sentence. The present disclosure is not limited to this.

[0060] Optionally, the semantics referred to by the second text feature (sentence feature) is the complete information conveyed by the entire sentence as a whole, that is, "a black car in motion" describes a state of a car of a specific color in motion. The present disclosure is not limited thereto.

[0061] Optionally, a preliminary overall semantic representation (hereinafter referred to as basic semantic features) can be extracted from the text data by using the RoBERT model. The RoBERT semantic feature extractor can capture the deep semantic relationship of the text, including context dependency and the correlation between words. On this basis, the basic semantic features can be mapped to the first text feature and the second text feature respectively to obtain text features at two different semantic levels. The present disclosure is not limited to this.

[0062] In operation S303, based on the video frame in the video data, a video feature corresponding to the video frame is determined, where the video feature includes a visual feature corresponding to each pixel in the video frame.

[0063] Optionally, the video feature refers to a feature extracted from a video frame in the video data, which is used to describe the visual information of each pixel in the video frame. For example, a convolutional neural network (CNN) can be used to process the video frame to generate a feature map representing the semantic and spatial content of the video frame. CNN is a deep learning model suitable for processing image and video data, which extracts local features through convolution operations and aggregates contextual information layer by layer. Take the tth frame I of the video t For example, CNN can generate the corresponding feature map in, C l , H l , W l Respectively represent the number of channels, height and width, l∈{1,2,3,4}. The present disclosure is not limited thereto.

[0064] Optionally, the visual feature corresponding to each pixel refers to the deep visual representation obtained at each pixel position on the feature map after being processed by the visual projection layer 401. Specifically, this process involves mapping each pixel point in the input image to a high-dimensional feature space through a series of calculations and transformations. In this feature space, each pixel point no longer represents only its original color or brightness information, but contains richer contextual information and semantic information, such as edge information, texture information, shape information, etc. The present disclosure is not limited to this.

[0065] In operation S304, the visual feature corresponding to each pixel in the video frame is fused with the first text feature to determine a first fused feature. In operation S305, the visual feature corresponding to each pixel in the video frame is fused with the second text feature to determine a second fused feature.

[0066] Optionally, fusion refers to the process of combining features of multiple modalities under certain rules to generate a comprehensive expression, that is, a fused feature. For example, fusion can be achieved through a convolution operation, in which features of different modalities are integrated into a new feature in a specific dimension. For the visual features and the first text features (word features) of each pixel in the video frame, fusion can be achieved by pixel-by-pixel weighted convolution or element-by-element weighted summation to generate a first fused feature containing multimodal information. Similarly, the second fused feature can also be fused in a similar manner. The present disclosure is not limited to this.

[0067] Thus, the first fused feature obtained is a joint feature expression obtained by fusing the visual feature and the word feature. Based on the visual feature of each pixel in the video frame, the word feature is introduced through operation S304 to generate a high-dimensional feature representation containing image and word-level semantics. The first fused feature can describe the visual information at the pixel level and also transmit the supplementary information at the word level from the text modality. Similarly, the second fused feature obtained is a joint feature expression obtained by fusing the visual feature and the sentence feature. Based on the visual feature of each pixel in the video frame, the sentence feature is introduced through operation S305 to generate a high-dimensional feature representation containing image and sentence-level semantics. The second fused feature can describe the visual information at the pixel level and also transmit the supplementary information at the sentence level from the text modality. The present disclosure is not limited to this.

[0068] In operation S306, an identification feature for identifying a target to be identified in the video data is determined based on the video feature, the first fusion feature, and the second fusion feature.

[0069] Optionally, the identification features used to identify the target to be identified in the video data are high-level feature expressions extracted from the video data and used for the RMOT task. For example, a multi-layer neural network is used to generate identification features that characterize the target shape, texture, dynamic trajectory, and semantic information in combination with video features, first fusion features, and second fusion features. These features contain both low-level visual details and integrate multimodal information in fusion features to support the implementation of RMOT tasks with higher performance. The present disclosure is not limited to this.

[0070] In operation S307, the target to be identified is identified in the video frame based on the identification feature.

[0071] Optionally, in operation S307, the recognition features may be further processed to locate and classify the target in the video frame. For example, for a vehicle recognition task in a video frame, the recognition features may be used to determine the location of vehicles that meet a specific description, and draw bounding boxes to mark the targets, such as identifying all black vehicles that meet a reference description. These bounding boxes are adjusted in real time according to the dynamic movement of the vehicle in the video frame to maintain the accuracy of target tracking. The present disclosure is not limited thereto.

[0072] Taking the video surveillance scene as an example, the vehicle can be identified and marked in the video frame based on the color, shape and motion characteristics of the vehicle, and the position and size of the bounding box will be dynamically adjusted as the vehicle changes in the scene. This recognition method is not only applicable to single-target scenes, but also to multi-target detection and tracking. Of course, the present disclosure is not limited to this.

[0073] Therefore, in one aspect of the present disclosure, the embodiments of the present disclosure achieve fine-grained fusion of pixel-level visual features and text features at multiple semantic levels, thereby being able to fully utilize text data and significantly improving the performance of the RMOT task.

[0074] Next reference Figure 4 To describe some optional details of the embodiments of the present disclosure. Figure 4 A schematic diagram of a multimodal fusion module 40 according to an embodiment of the present disclosure is shown, which can be used to implement some operations according to the embodiment of the present disclosure, such as operations S302 to S306. Of course, the present disclosure is not limited thereto.

[0075] According to the multimodal fusion module 40 of the embodiment of the present disclosure, it can be used to fuse the features of two different modes, namely, visual features and text features, in a fine-grained manner, and process them at two semantic levels, namely, word level and sentence level. The multimodal fusion module 40 includes a visual projection layer 401 (Visual Project Layer, VPL), a text projection layer 402 (Text Project Layer, TPL), a token partition module 403 (token partition module), a word-level fusion module 404 (word-level fusion module), a sentence-level fusion module 405 (sentence-level fusion module) and an aggregation module 406 (Aggregation module). Of course, the multimodal fusion module 40 can also include more modules or fewer modules, and the present disclosure is not limited thereto.

[0076] For the convenience of explanation, it is assumed that in operation S302, basic semantic features are extracted from text data by using the RoBERT model, and the basic semantic features include multiple word units. The basic semantic features are represented by S∈RL×D In other words, L represents the number of words in the text, and D is the feature dimension of the word vector. It is assumed that the video frame I is obtained by using CNN in operation S303. t The visual feature map of in, Here, the parameter l∈{1,2,3,4} is used to identify the visual feature map F t Each level corresponds to a feature map with a different semantic abstraction level, and the resolution and number of feature channels of the feature map will change with the level. The present disclosure is not limited to this.

[0077] Channels are the number of feature dimensions contained in the feature map. In the original input, the number of channels can correspond to the color channels of the image (such as the three channels of an RGB image), and after CNN extraction, the number of channels in the feature map C l It reflects the dimension or type of features extracted in this layer. The height H of the feature map l and width W l Then they represent the resolution of the spatial dimension respectively. By adjusting the model structure, parameters such as the number of channels and feature map size can be adapted to different task requirements. The present disclosure is not limited to this.

[0078] In one aspect of the present disclosure, Figure 4 As shown, in order to align the visual feature map F t Channels with basic semantic features S, visual feature maps After being processed by the independent visual projection layer 401, the number of channels is reduced to d, and we get At the same time, the basic semantic feature S is mapped to S′∈R through the text projection layer 402 L×d , wherein the visual projection layer 401 is composed of a 1×1 convolution layer and a normalization layer, and the text projection layer 402 is composed of a fully connected layer and a normalization layer. Of course, the present disclosure is not limited to this.

[0079] Next, in operation S302, the set of the second word to the penultimate word in the basic semantic feature can be divided into the first text feature, and the first word in the basic semantic feature can be divided into the second text feature. Of course, the present disclosure is not limited to this.

[0080] For example, the first word in the basic semantic feature (or the basic semantic feature processed by the text projection layer 402) represents the semantics of the entire sentence, the last word represents the end of the sentence, and the remaining words correspond to words in the text data. Based on this, the word-unit division module 403 can divide the basic semantic feature S into word features S′ w ∈R (L-2)×d Sentence feature S′ s ∈R 1×d .

[0081] The word-level fusion module 404 and the sentence-level fusion module 405 use the word feature S′ w Sentence feature S′ s With the first layer of visual features Fusion is performed to generate the first fusion feature and the second fusion feature

[0082] For example, in operation S304, in order to generate the first fusion feature The word-level fusion module 404 can perform convolution operations on the visual features corresponding to each pixel in the video frame and the feature values ​​corresponding to each word in the first text feature, respectively, to generate multiple intermediate features corresponding to the pixel, wherein each intermediate feature corresponds to a word in the first text feature, and select the maximum value from the multiple intermediate features corresponding to each pixel in the video frame as the word-level fusion feature corresponding to the pixel, and use the set of word-level fusion features corresponding to all pixels as the first fusion feature. The specific calculation formula can be shown as follows.

[0083]

[0084] Among them, i,j are feature maps , c is the number of channels, and k is the kth word in the input text. Thus, the word-level fusion module 404 can be based on the semantic information S of each word. ′ w (k,c) to respectively analyze the visual features of each pixel Modulation to generate multiple intermediate features The size of these intermediate features characterizes the association between each word and a specific pixel. Therefore, by using multiple intermediate features F′ corresponding to each pixel in the video frame vw Selecting the maximum value among (i, j, k) can find the specific word with the greatest correlation with each pixel in the text data. Therefore, this maximum intermediate feature is used as the word-level fusion feature F corresponding to the pixel at (i, j). vw (i, j) can better improve the fusion degree of text data with video data in word-level semantics and better utilize the semantic information in text data. Of course, the present disclosure is not limited to this.

[0085] For example, in operation S305, in order to generate the second fusion feature The sentence-level fusion module 405 may perform a convolution operation on the visual feature corresponding to each pixel in the video frame and the second text feature pixel by pixel to determine the second fusion feature. The specific calculation formula may be shown as follows.

[0086]

[0087] Among them, i,j are feature maps The horizontal and vertical coordinates of are, and c is the number of channels. Of course, the present disclosure is not limited thereto.

[0088] For example, in operation S305, the aggregation module 406 may superimpose the video feature with the first fusion feature and the second fusion feature by channel to generate a preliminary recognition feature; and convert the preliminary recognition feature into a two-dimensional tensor, which is used as a recognition feature for identifying the target to be recognized in the video data. vw and F vs Add channel by channel On the top, the preliminary recognition features are generated by formula (4)

[0089]

[0090] In this way, the aggregation module 406 can efficiently integrate the four-layer visual features The fusion information of the text feature S at the word level and sentence level. To further enhance the recognition features, the recognition features of the four layers can be further flattened and spliced ​​into a two-dimensional tensor (the two-dimensional tensor is also the recognition feature after the enhancement of the preliminary recognition features), and then input into the target recognition module for subsequent processing. Of course, the present disclosure is not limited to this.

[0091] Therefore, the hierarchical multimodal fusion module 40 of the embodiment of the present disclosure realizes the fine-grained fusion of pixel-level visual features and text features at multiple semantic levels. Therefore, the embodiment of the present disclosure can fully apply text data and greatly improve the performance of the RMOT task.

[0092] Next, refer to Figure 5 To describe some optional details of the embodiments of the present disclosure. Figure 5 The overall architecture diagram of the neural network model 50 for performing the RMOT task according to the embodiment of the present disclosure is shown. It can be used to implement all operations according to the embodiment of the present disclosure. Of course, the present disclosure is not limited to this.

[0093] Specifically, the neural network model 50 for performing the RMOT task includes but is not limited to: a video frame feature extraction module 501, a text feature extraction module 502, a multimodal fusion module 40, and a target recognition module 503. The neural network model 50 for performing the RMOT task may also include an acquisition module (not shown) for performing operation S301. Among them, the video frame feature extraction module 501 can be used to perform operation S303, and the text feature extraction module 502 can be used to extract basic semantic features.

[0094] For example, we can assume that the input obtained by the acquisition module is a (video, text) pair, and the sequence length of the video video is N, which is recorded as video = {I1, I2, ..., I N}, the corresponding text data is recorded as text. The model processes each video in time sequence. At timestamp t, the video frame feature extraction module 501 can be used to extract the tth frame I in the video video. t Extract feature map At the same time, the text feature extraction module 502 is used to extract text features S from the input text. Then, the multimodal fusion module 40 is used to extract the text features S from each layer of feature maps F. t The fine-grained fusion of the text feature S generates a highly expressive visual-text fusion feature, thereby aligning all semantically matching objects in the video. Figure 4 The multimodal fusion module 40 has been described in detail, and the present disclosure will not repeat it here.

[0095] Next, the target recognition module 503 for performing operation S307 is further described. The target recognition module 503 may include but is not limited to: an encoder 5031 (encoder), a decoder 5032 (Decoder), a temporal enhancement module 5033 (Temporal Enhancement Module) and a prediction head 5034 (Predict Head). Of course, the present disclosure is not limited to this.

[0096] Optionally, in operation S307, the target recognition module 503 may encode the recognition feature using an encoder 5031 to obtain the encoded recognition feature; based on the basic semantic features and the query features, the encoded recognition feature is decoded using a decoder 5032 to obtain the embedding of the target to be identified. Then, based on the embedding of the target to be identified, the target to be identified is identified in the video frame. The embedding of the target to be identified will include information pointing to the target to be identified, location information of the target to be identified, and category information of the target to be identified. Of course, the present disclosure is not limited to this.

[0097] Thus, in operation S307, the target recognition module 503 can input the recognition features determined by the multimodal fusion module 40 into the encoder 5031, thereby further enhancing the interaction between the image and text features, effectively improving the expressiveness of the visual text features, and thus obtaining the encoded recognition features; then, in combination with the basic semantic features and the query features, the decoder 5032 is enabled to decode the encoded recognition features to obtain the embedding of the target to be recognized; finally, based on the embedding of the target to be recognized, the target to be recognized is accurately recognized in the video frame. The present disclosure is not limited thereto.

[0098] For example, in a neural network, the encoder 5031 can be used to compress and encode input data, extract key information, and convert it into a specific intermediate representation; the decoder 5032 receives the intermediate result output by the encoder 5031 and reversely restores the data that matches the target task, such as text generation and image reconstruction. The two work together to achieve complex information processing and conversion.

[0099] For example, in one aspect of the present disclosure, the decoder 5032 optionally uses an attention mechanism and a text guidance mechanism, aiming to focus on the key parts of the encoded recognition features through basic semantic features and query features, assign different weights to different positions, and highlight key information. Specifically, the query feature used by the decoder 5032 is a given kind of guidance information, similar to a search clue, which guides the decoder 5032 to obtain matching feature information in the video frame, and assists the decoder 5032 in obtaining accurate and precise positioning and identifying targets. The combined use of these technologies enables the target recognition module 40 to better combine the semantic information in the text data to accurately identify the target to be identified. The present disclosure is not limited to this.

[0100] Optionally, the query features include detection query features (detect queries) and tracking query features (track queries), the detection query features are randomly initialized before being input into the decoder 5032, and after being decoded by the decoder 5032, they become embeddings for detecting the target to be identified that did not appear in the previous frame, and the tracking query features are embeddings for detecting the target to be identified that appeared in the previous frame before being input into the decoder 5032, and after being decoded by the decoder 5032, they become embeddings for detecting the target to be identified that appeared in the previous frame. Of course, the present disclosure is not limited to this.

[0101] Optionally, the detection query feature is a learnable feature. That is, the initial state of the detection query feature is a randomized feature, which gradually evolves into a decoded feature containing contextual semantic information through multiple iterations of the decoder 5032. The detection query feature is transformed from a randomized feature into a feature that integrates key information of multiple features (including tracking query features, identification features, and basic semantic features) through multiple iterations of the decoder 5032, and finally obtains the ability to match input text data. Of course, the present disclosure is not limited to this.

[0102] Optionally, the tracking query feature refers to the tracking query feature of the previous timestamp, which is mainly used to associate information across frames. The tracking query feature can be provided by the timing enhancement module 5033. In the decoder 5032, after being input together with other features, as a key component of the timing information, the decoder 5032 is supported to track the object to be identified across frames, thereby improving the ability of the neural network model used to perform the RMOT task to aggregate historical information between multiple frames. Of course, the present disclosure is not limited to this.

[0103] Optionally, the prediction head 5034 can be used to identify the target to be identified in the video frame based on the embedding of the target to be identified. The embedding of the target to be identified is input into the prediction head 5034, and the appearance of the target to be identified in the current video frame is predicted based on the text data. Of course, the present disclosure is not limited to this.

[0104] Thus, in one aspect, the various modules in the neural network model for performing the RMOT task according to the embodiment of the present disclosure can work together to effectively achieve the goal of the RMOT task and ensure that the model accurately identifies and tracks objects that match the text data in complex video scenes.

[0105] Next, refer to Figure 6 To describe some optional details of the embodiments of the present disclosure. Figure 6 A schematic diagram of a decoder of an improved neural network model according to an embodiment of the present disclosure is shown.

[0106] like Figure 6As shown, the query features input to the text-guided decoder are divided into detection query features and tracking query features. The detection query features are randomly initialized before the text-guided decoder is input, and after being decoded by the text-guided decoder, they are converted into embeddings for detecting the target to be identified that did not appear in the previous frame, and then the embeddings for detecting the target to be identified that did not appear in the previous frame can be used to generate the bounding box and category information of the instance corresponding to the target to be identified that did not appear in the previous frame. The tracking query feature is the embedding output by the decoder that is passed down from the previous frame. The embedding indicates the target to be identified that has appeared in the previous frame and is stored in the timing enhancement module 5033. It is updated in the current frame to continuously track the same instance. The tracking query feature acts as a carrier of the target to be queried in the target trajectory, directly affecting the final prediction result.

[0107] Optionally, the decoder can be a text-guided decoder. A text-guided decoder is a decoder structure that combines text information to guide target feature extraction, and injects semantic information (i.e., basic semantic features) into the decoding process to help improve the semantic relevance and contextual adaptability of feature expression. Specifically, when executing "decoding the encoded recognition features based on the basic semantic features and query features", the decoder can perform: based on the basic semantic features, modulating the query features to obtain text-guided query features; and based on the encoded recognition features, modulating the text-guided query features to obtain the embedding of the target to be identified. Of course, the present disclosure is not limited to this.

[0108] Optionally, the text-guide decoder of the embodiment of the present disclosure is composed of N pairs of decoding layers 50321 and text-guide modules 50322 (text-guide modules), which can realize semantic perception of query features (including detecting query features and tracking query features). Of course, the present disclosure is not limited to this.

[0109] Optionally, the text guidance module can be composed of a self-attention layer 503221 and a text-to-query cross-attention module. In the self-attention layer, query features (including detection query features and tracking query features) are subjected to a self-attention operation to capture the correlation between different positions in the features. Self-attention enhances the expressiveness of information by giving higher weights to important parts. The query features after the self-attention operation are optionally processed by normalization and residual connection, wherein normalization is a standardized operation used to stabilize feature distribution; residual connection improves the gradient flow of model training by retaining the information flow between input and output. The present disclosure is not limited to this.

[0110] For example, the query vector result after normalization and introduction of residual connection can be used as a query for a cross-attention operation, and the cross-attention operation is performed with the basic semantic features as key features and value features. The cross-attention layer is intended to extract contextual information related to the task through the interaction between the query features and the basic semantic features. After the cross-attention operation, the result is normalized and residually connected again to further stabilize the feature representation and retain the input features, thereby providing a query input for the decoding layer paired with the text-guided module, which is hereinafter referred to as a text-guided query feature. The present disclosure is not limited to this.

[0111] For example, each decoding layer may include three parts: a self-attention layer, a multi-scale deformable cross-attention (MSDeformable cross-attention) layer, and a feed-forward neural network layer. Similar to the self-attention layer of the text-guided module, the self-attention layer of the decoder may perform a self-attention operation on the text-guided query. Next, the multi-scale deformable cross-attention layer may use the encoded recognition features as key features and value features, and the features output by the self-attention layer of the decoder as query features, so as to associate the features output by the self-attention layer of the decoder and the encoded recognition features at different scales, so that the decoding layer can better adapt to changes in the target area. The output of the multi-scale deformable cross-attention layer will be input to the feed-forward neural network layer, which is a fully connected network composed of a series of linear transformations and activation functions, for further extraction and nonlinear transformation feature representation. Of course, the present disclosure is not limited thereto.

[0112] The embedding of the target to be identified after being processed by the decoder will be input and stored in the timing enhancement module 5033 as the tracking query feature of the next frame. Therefore, in some aspects of the present disclosure, this step can be used to enhance the temporal consistency of the query feature, thereby further improving the correlation ability of the query feature between video frames, and providing stronger support for subsequent target tracking and prediction.

[0113] Therefore, in an optional aspect, an embodiment of the present disclosure utilizes text to guide the decoder, providing a priori semantic information (basic semantic features) for the query during the decoding process, so that the decoder can more fully utilize the semantic information in the text data and improve the embedding performance of the target to be identified.

[0114] Next, refer to Figure 7 To describe some optional details of the embodiments of the present disclosure. Figure 7 A schematic diagram of a prediction head of an improved neural network model according to an embodiment of the present disclosure is shown.

[0115] Optionally, the prediction head of the improved neural network model according to the embodiment of the present disclosure may be a text-guided prediction head (Text-Guide Predict Head, TGPH). Optionally, according to the embodiment of the present disclosure, the prediction head of the improved neural network model may be a text-guided prediction head (Text-Guide Predict Head, TGPH). The text-guided prediction head is a module that combines text information to guide target prediction, aiming to achieve comprehensive prediction of the target to be identified through category, border and pointing information. TGPH consists of category branch, border branch and pointing branch, and the specific structure is as follows Figure 7 As shown. The embedding of the target to be identified after being processed by the text-guided decoder will be used as input for TGPH prediction to determine the spatial coordinate information of the target to be identified in the video frame, the confidence information of whether the object at the position corresponding to the spatial coordinate information is the target to be identified, and the probability that the object at the position corresponding to the spatial coordinate information matches the text data. In this way, the target to be identified is "identified" in the video frame. Of course, the present disclosure is not limited to this.

[0116] For example, the category branch of TGPH consists of a fully connected layer and a Sigmoid activation function, which outputs a confidence score to indicate whether the query represents a real object or an empty object. The category branch can effectively distinguish the target object from the non-target area, ensuring that the predicted category information has a high accuracy. Of course, the present disclosure is not limited to this.

[0117] Optionally, the border branch is composed of a four-layer feedforward neural network, and all layers except the last layer use a ReLU activation function, which is intended to predict the position and size of the border of the target to be identified. By extracting the spatial features of the target layer by layer, this branch can accurately generate the spatial coordinate information of the target to be identified. The border branch provides necessary support for the positioning of the target to be identified, thereby ensuring that the detection result is consistent with the actual target position in the video frame. Of course, the present disclosure is not limited to this.

[0118] For example, the referent branch first receives the first text feature and the second text feature, processes the embedding of the first text feature and the target to be identified through the cross attention mechanism, and generates a cross attention result. That is, the first text feature is used as the key feature and the value feature, and the embedding of the target to be identified is used as the query feature to generate a cross attention result. Then, the second text feature, the embedding of the target to be identified, and the cross attention result are combined by element-by-element addition, and the synthesized result is input into the fully connected layer and a referent score is generated by the Sigmoid activation function. The referent score indicates the probability of the instance matching the text description, and provides a guiding basis for trajectory generation and multi-target matching. Of course, the present disclosure is not limited to this.

[0119] Next, refer to Figure 8 To describe some optional details of the embodiments of the present disclosure. Figure 8 A schematic diagram of an improved neural network model training method 80 according to an embodiment of the present disclosure is shown.

[0120] The improved neural network model training method 80 according to the embodiment of the present disclosure includes the following operations S801 to S808. Of course, the method 80 may also include more operations or fewer operations, and the present disclosure is not limited thereto.

[0121] In operation S301, a video sample and a text sample corresponding to the video sample are obtained, wherein the text sample indicates a target to be identified in the video sample, and the video sample includes a plurality of video frames arranged in time sequence. Optionally, the definitions of the video sample and the text sample refer to the video data and text data described in the foregoing, and the only difference is that the video sample and the text sample are used for model training. Therefore, the video sample also has corresponding annotation data, that is, indicating the true value of the target to be identified in each frame of the video sample. Of course, the present disclosure is not limited to this.

[0122] In operation S302, based on the text sample, the multimodal fusion module is used to determine a first text feature and a second text feature corresponding to the text sample, wherein the first text feature and the second text feature correspond to different semantic levels. The multimodal fusion module has been described with reference to the preceding figures, and the disclosure will not be repeated here.

[0123] In operation S303, based on the video frame in the video sample, a video feature corresponding to the video frame is determined, where the video feature includes a visual feature corresponding to each pixel in the video frame.

[0124] In operation S304, the multimodal fusion module is used to fuse the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature.

[0125] In operation S305, the multimodal fusion module is used to fuse the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature.

[0126] In operation S306, the multimodal fusion module is used to determine an identification feature for identifying a target to be identified in the video sample based on the video feature, the first fusion feature, and the second fusion feature.

[0127] In operation S307, based on the recognition feature, the target recognition module is used to determine a prediction value of the target to be recognized in the video frame.

[0128] In operation S308, at least one of the multimodal fusion module and the target recognition module is trained based on the difference between the predicted value and the true value of the target to be recognized.

[0129] Optionally, in one aspect, an embodiment of the present disclosure adopts a look-back training strategy to optimize the utilization efficiency of training data and promote the learning of difficult samples.

[0130] There are two major problems with traditional training strategies: First, some frames of the video sample may not contain positive samples, because the target to be identified mentioned in the text data often does not run through the entire video sample, which makes the randomly paired samples lack positive samples corresponding to the text data. Second, since each video may be associated with multiple text data, for example, some text data may be omitted during the training process. This results in some (video sample, text sample) pairs not being used throughout the training process, missing learning opportunities, and affecting the robustness and performance of the model.

[0131] The embodiment of the present disclosure solves this problem through a backtracking training strategy, including two sub-strategies: the backtracking hardening strategy (Look-Back-Hard) and the backtracking remaining strategy (Look-Back-Remain), so as to more efficiently utilize the labeled data, thereby improving the training effect and model performance. Of course, the present disclosure is not limited to this.

[0132] For example, when using the backtracking reinforcement strategy to train a neural network, in the early stage of model training, we can focus on mining sample pairs with large differences between the predicted value and the true value of the target to be identified in the training samples, and use these difficult samples to feed back the model learning process. Of course, the present disclosure is not limited to this.

[0133] For example, when a neural network is trained using a backtracking reinforcement strategy, the following steps may be performed: in a preset training cycle in the early stage of training, based on the difference between the predicted value and the true value of the target to be identified, at least one of the multimodal fusion module and the target identification module is trained, and the difference between the predicted value and the true value corresponding to each training sample pair is recorded; at the end of the preset training cycle in the early stage of training, multiple pairs of video samples and text samples with the highest difference between the predicted value and the true value are selected as high-loss sample sets; and the high-loss sample set is used to train at least one of the multimodal fusion module and the target identification module. Of course, the present disclosure is not limited to this.

[0134] In the early stage of model training, the backtracking reinforcement strategy optimizes the learning process of the model by identifying and utilizing training sample pairs (hereinafter referred to as high-loss sample pairs) with large differences between the predicted value and the true value of the target to be identified. These high-loss samples usually contain high-quality annotated data that is highly consistent with the text data target. The embodiment of the present disclosure adopts an adaptive method to select these samples for retraining within a specified training cycle. Specifically, within a preset training cycle (for example, the sth training cycle to the eth training cycle), the model can be standardly trained according to a conventional strategy, while recording the difference between the predicted value and the true value of the target to be identified corresponding to each sample pair. At the end of each cycle, the top 30% of the samples with the largest difference between the predicted value and the true value of the target to be identified are selected as high-loss sample sets for additional training to ensure that valuable data is not ignored during the training process. Of course, the present disclosure is not limited to this.

[0135] For example, the backtracking residual strategy is a strategy for re-incorporating (video sample, text sample) pairs (hereinafter referred to as residual sample sets) that are not enabled in the regular training process into the training scope at a specific training cycle node and giving them additional training. Of course, the present disclosure is not limited to this.

[0136] For example, when a neural network is trained using a backtracking residual strategy, the following steps may be performed: in a training cycle at a later stage of training, a plurality of paired video samples and text samples that are not used for training are determined as a residual sample set; and at least one of the multimodal fusion module and the target recognition module is trained using the residual sample set. Of course, the present disclosure is not limited thereto.

[0137] The backtracking residual strategy provides an additional training method for some unused (video samples, text samples) pairs. Specifically, in each training cycle, the sample pairs that are not used for training are recorded, and these samples are used for additional training at the end of the cycle. The embodiment of the present disclosure applies this strategy only in the last cycle of model training, significantly improving the robustness and performance of the model at the lowest training cost. The combination of these two strategies not only solves the problem of insufficient data utilization in existing training methods, but also effectively enhances the training effect and final performance of the model. Of course, the present disclosure is not limited to this.

[0138] Optionally, these two backtracking training sub-strategies cooperate with each other to form a complete backtracking training strategy system. The backtracking reinforcement strategy focuses on the early mining of high-value difficult samples, allowing the model to lay a solid foundation in the basic training stage and quickly adjust parameters to adapt to complex sample situations; while the backtracking residual strategy checks for leaks and catches unused sample resources at the end of training to further polish model performance. For example, in actual training scenarios, the backtracking reinforcement strategy is used in the early stage to prompt the model to accurately lock in error-prone and high-loss samples and enhance the ability to recognize complex scenarios; the later backtracking residual strategy maximizes the value of the missing sample data and fills the model's shortcomings. Of course, the present disclosure is not limited to this.

[0139] Optionally, the advantage of the backtracking reinforcement strategy is also reflected in its flexibility in dealing with data sets of different sizes. When the data set is small, the backtracking reinforcement strategy can fully extract the learning value of limited samples to avoid insufficient model training due to data scarcity; at the same time, the backtracking residual strategy can efficiently sort out unused sample pairs to ensure that massive data is not missed. For example, in a small video annotation data set, the backtracking reinforcement strategy repeatedly refines key samples to improve model accuracy; the backtracking residual strategy explores hidden value and enhances model robustness. Of course, the present disclosure is not limited to this.

[0140] According to yet another aspect of the present disclosure, an electronic device is provided for implementing the method according to the embodiment of the present disclosure. Fig. 9 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0141] like Fig. 9 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 2010, the method described above may be executed.

[0142] The processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may be an X86 architecture or an ARM architecture.

[0143] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0144] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig.10 The architecture of the computing device 3000 shown in FIG. Fig.10 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Fig.10 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.10 One or more components of a computing device are shown.

[0145] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Fig.11 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0146] like Fig.11As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method according to the embodiment of the present disclosure.

[0148] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0150] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure. For example, an overview of some aspects of the present disclosure is provided below, and these aspects may be combined with any other aspect.

[0151] Aspect 1: A method for processing video data is provided, the method comprising: acquiring video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data comprises a plurality of video frames arranged in time sequence; determining a first text feature and a second text feature corresponding to the text data based on the text data, wherein the semantic levels corresponding to the first text feature and the second text feature are different; determining a video feature corresponding to the video frame based on the video frame in the video data, the video feature comprising a visual feature corresponding to each pixel in the video frame; fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; fusing the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; determining an identification feature for identifying a target to be identified in the video data based on the video feature, the first fused feature and the second fused feature; and identifying the target to be identified in the video frame based on the identification feature.

[0152] Aspect 2: The first text feature is a word feature, the second text feature is a sentence feature, and determining the first text feature and the second text feature corresponding to the text data based on the text data includes: determining a basic semantic feature based on the text data, the basic semantic feature including multiple word units; dividing the set from the second word unit to the second to last word unit in the basic semantic feature into a first text feature; and dividing the first word unit in the basic semantic feature into a second text feature.

[0153] Aspect 3: The step of fusing the visual features corresponding to each pixel in the video frame with the first text feature to determine the first fused feature includes: performing convolution operations on the visual features corresponding to each pixel in the video frame with the feature values ​​corresponding to each word in the first text feature, pixel by pixel, to generate a plurality of intermediate features corresponding to the pixel, wherein each intermediate feature corresponds to a word in the first text feature, and selecting a maximum value from the plurality of intermediate features corresponding to each pixel in the video frame as the word-level fused feature corresponding to the pixel, and taking the set of word-level fused features corresponding to all pixels as the first fused feature.

[0154] Aspect 4: Fusion of the visual feature corresponding to each pixel in the video frame with the second text feature to determine the second fused feature includes: convolution operation of the visual feature corresponding to each pixel in the video frame with the second text feature pixel by pixel to determine the second fused feature.

[0155] Aspect 5: Determining the identification feature for identifying the target to be identified in the video data based on the video feature, the first fusion feature and the second fusion feature includes: superimposing the video feature with the first fusion feature and the second fusion feature by channel to generate a preliminary identification feature; and converting the preliminary identification feature into a two-dimensional tensor, which serves as the identification feature for identifying the target to be identified in the video data.

[0156] Aspect 6: The step of identifying the target to be identified in the video frame based on the identification feature includes: encoding the identification feature using an encoder to obtain the encoded identification feature; decoding the encoded identification feature using a decoder based on the basic semantic feature and the query feature to obtain the embedding of the target to be identified; and identifying the target to be identified in the video frame based on the embedding of the target to be identified.

[0157] Aspect 7: The query features include detection query features and tracking query features. The detection query features are randomly initialized before being input into the decoder, and after being decoded by the decoder, they become embeddings for detecting the target to be identified that did not appear in the previous frame. The tracking query features are embeddings of the target to be identified that appeared in the previous frame before being input into the decoder, and after being decoded by the decoder, they become embeddings for detecting the target to be identified that appeared in the previous frame.

[0158] Aspect 8: The decoding of the encoded recognition features using a decoder based on the basic semantic features and the query features includes: modulating the query features based on the basic semantic features to obtain text-guided query features; and modulating the text-guided query features based on the encoded recognition features to obtain the embedding of the target to be identified.

[0159] Aspect 9: Based on the embedding of the target to be identified, identifying the target to be identified in the video frame includes: based on the embedding of the target to be identified, determining the spatial coordinate information of the target to be identified in the video frame, the confidence information of whether the object at the position corresponding to the spatial coordinate information is the target to be identified, and the probability that the object at the position corresponding to the spatial coordinate information matches the text data.

[0160] Aspect 10: A method for training a neural network is provided, wherein the neural network model includes a multimodal fusion module and a target recognition module, wherein the method includes: obtaining a video sample and a text sample corresponding to the video sample, wherein the text sample indicates a target to be recognized in the video sample, and the video sample includes a plurality of video frames arranged in time sequence; based on the text sample, using the multimodal fusion module, determining a first text feature and a second text feature corresponding to the text sample, wherein the first text feature and the second text feature correspond to different semantic levels; based on the video frame in the video sample, determining a video feature corresponding to the video frame, wherein the video feature includes a visual feature corresponding to each pixel in the video frame; using the The multimodal fusion module fuses the visual feature corresponding to each pixel in the video frame with the first text feature to determine the first fusion feature; the multimodal fusion module fuses the visual feature corresponding to each pixel in the video frame with the second text feature to determine the second fusion feature; the multimodal fusion module determines the identification feature for identifying the target to be identified in the video sample based on the video feature, the first fusion feature and the second fusion feature; based on the identification feature, the target identification module determines the predicted value of the target to be identified in the video frame; and based on the difference between the predicted value and the true value of the target to be identified, trains at least one of the multimodal fusion module and the target identification module.

[0161] Aspect 11: The training of at least one of the multimodal fusion module and the target recognition module includes: within a preset training cycle in the early stage of training, based on the difference between the predicted value and the true value of the target to be identified, training at least one of the multimodal fusion module and the target recognition module, and recording the difference between the predicted value and the true value corresponding to each training sample pair; at the end of the preset training cycle in the early stage of training, selecting multiple pairs of video samples and text samples with the highest difference between the predicted value and the true value as a high-loss sample set; and using the high-loss sample set to train at least one of the multimodal fusion module and the target recognition module.

[0162] Aspect 12: The training of at least one of the multimodal fusion module and the target recognition module includes: determining, in a training cycle in a later stage of training, a plurality of pairs of video samples and text samples that have not been used for training as a residual sample set; and using the residual sample set to train at least one of the multimodal fusion module and the target recognition module.

[0163] Aspect 13: A device for processing video data, the device comprising: an acquisition module, configured to: acquire video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data comprises a plurality of video frames arranged in time sequence; a video frame feature extraction module, configured to: determine, based on the video frames in the video data, the video features corresponding to the video frames, the video features comprising visual features corresponding to each pixel in the video frames; a multimodal fusion module, configured to: determine, based on the text data, a first text feature and a second text feature corresponding to the text data, wherein the first text feature and the second text feature correspond to different semantic levels; fuse the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; fuse the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; determine an identification feature for identifying a target to be identified in the video data based on the video feature, the first fused feature and the second fused feature; and a target identification module, configured to: identify the target to be identified in the video frame based on the identification feature.

[0164] Aspect 14: An electronic device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the method described in any aspect is executed.

[0165] Aspect 15: A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method described in any aspect is implemented.

Claims

1. A method for processing video data, the method comprising: Acquire video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data includes a plurality of video frames arranged in time sequence; Based on the text data, determining a first text feature and a second text feature corresponding to the text data, wherein the first text feature and the second text feature correspond to different semantic levels; Based on a video frame in the video data, determining a video feature corresponding to the video frame, the video feature comprising a visual feature corresponding to each pixel in the video frame; Fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; Fusing the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; Determining, based on the video feature, the first fusion feature, and the second fusion feature, an identification feature for identifying a target to be identified in the video data; and Based on the identification features, the target to be identified is identified in the video frame.

2. The method for processing video data according to claim 1, wherein: The first text feature is a word feature, the second text feature is a sentence feature, and the determining, based on the text data, the first text feature and the second text feature corresponding to the text data comprises: Based on the text data, determining basic semantic features, the basic semantic features including a plurality of word units; dividing a set of the second word unit to the second to last word unit in the basic semantic feature into a first text feature; and The first word in the basic semantic feature is divided into a second text feature.

3. The method for processing video data according to claim 2, wherein: The fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine the first fused feature comprises: Convolution operations are performed on the visual features corresponding to each pixel in the video frame with the feature values ​​corresponding to each word in the first text feature, to generate a plurality of intermediate features corresponding to the pixel, wherein each intermediate feature corresponds to a word in the first text feature, and A maximum value is selected from a plurality of intermediate features corresponding to each pixel in the video frame as the word-level fusion feature corresponding to the pixel, and a set of word-level fusion features corresponding to all pixels is taken as the first fusion feature.

4. The method for processing video data according to claim 2, wherein: Fusion of the visual feature corresponding to each pixel in the video frame with the second text feature to determine the second fused feature comprises: A convolution operation is performed pixel by pixel on the visual feature corresponding to each pixel in the video frame and the second text feature to determine a second fusion feature.

5. The method for processing video data according to claim 1, wherein: The determining, based on the video feature, the first fusion feature, and the second fusion feature, of an identification feature for identifying a target to be identified in the video data comprises: superimposing the video feature with the first fused feature and the second fused feature by channel to generate a preliminary recognition feature; and The preliminary recognition feature is converted into a two-dimensional tensor, and the two-dimensional tensor is used as a recognition feature for identifying a target to be recognized in the video data.

6. The method for processing video data according to claim 2, wherein: The identifying the target to be identified in the video frame based on the identification feature comprises: Encoding the identification feature using an encoder to obtain an encoded identification feature; Based on the basic semantic features and the query features, using a decoder, decoding the encoded recognition features to obtain an embedding of the target to be recognized; and Based on the embedding of the to-be-recognized object, the to-be-recognized object is recognized in the video frame.

7. The method for processing video data according to claim 6, wherein: The query features include detection query features and tracking query features. The detection query feature is randomly initialized before being input into the decoder, and after being decoded by the decoder, it becomes an embedding for detecting the target to be identified that did not appear in the previous frame. The tracking query feature is the embedding of the target to be identified that has appeared in the previous frame before being input into the decoder, and becomes the embedding for detecting the target to be identified that has appeared in the previous frame after being decoded by the decoder.

8. The method for processing video data according to claim 7, wherein: The decoding of the encoded recognition feature by using a decoder based on the basic semantic feature and the query feature includes: Based on the basic semantic features, modulating the query features to obtain text-guided query features; and Based on the encoded recognition features, the text-guided query features are modulated to obtain an embedding of a target to be recognized.

9. The method for processing video data according to claim 6, wherein: The step of identifying the target to be identified in the video frame based on the embedding of the target to be identified includes: Based on the embedding of the target to be identified, determine the spatial coordinate information of the target to be identified in the video frame, the confidence information of whether the object at the position corresponding to the spatial coordinate information is the target to be identified, and the probability that the object at the position corresponding to the spatial coordinate information matches the text data.

10. A method for training a neural network, wherein the neural network model includes a multimodal fusion module and a target recognition module, wherein: The method comprises: Acquire a video sample and a text sample corresponding to the video sample, wherein the text sample indicates a target to be identified in the video sample, and the video sample includes a plurality of video frames arranged in time sequence; Based on the text sample, using the multimodal fusion module, determining a first text feature and a second text feature corresponding to the text sample, wherein the first text feature and the second text feature correspond to different semantic levels; Based on the video frame in the video sample, determining a video feature corresponding to the video frame, the video feature including a visual feature corresponding to each pixel in the video frame; Using the multimodal fusion module, the visual feature corresponding to each pixel in the video frame is fused with the first text feature to determine a first fused feature; Using the multimodal fusion module, the visual feature corresponding to each pixel in the video frame is fused with the second text feature to determine a second fused feature; Determining, by using the multimodal fusion module, an identification feature for identifying a target to be identified in the video sample based on the video feature, the first fusion feature, and the second fusion feature; Based on the recognition feature, using the target recognition module, determining a predicted value of the target to be recognized in the video frame; and Based on the difference between the predicted value and the true value of the target to be identified, at least one of the multimodal fusion module and the target identification module is trained.

11. The method of claim 10, wherein: The training of at least one of the multimodal fusion module and the target recognition module comprises: In a preset training cycle in the early stage of training, based on the difference between the predicted value and the true value of the target to be identified, at least one of the multimodal fusion module and the target identification module is trained, and the difference between the predicted value and the true value corresponding to each training sample pair is recorded; At the end of a preset training cycle in the early stage of the training, a plurality of pairs of video samples and text samples with the highest differences between the predicted values ​​and the true values ​​are selected as a high-loss sample set; and Using the high-loss sample set, at least one of the multimodal fusion module and the target recognition module is trained.

12. The method of claim 10, wherein: The training of at least one of the multimodal fusion module and the target recognition module comprises: In a training cycle at a later stage of training, a plurality of pairs of video samples and text samples that are not used for training are determined as a residual sample set; and At least one of the multimodal fusion module and the target recognition module is trained using the remaining sample set.

13. A device for processing video data, the device comprising: An acquisition module is configured to: acquire video data and text data corresponding to the video data, wherein the text data indicates a target to be identified in the video data, and the video data includes a plurality of video frames arranged in time sequence; A video frame feature extraction module is configured to: determine, based on a video frame in the video data, a video feature corresponding to the video frame, wherein the video feature includes a visual feature corresponding to each pixel in the video frame; The multimodal fusion module is configured as follows: Based on the text data, determining a first text feature and a second text feature corresponding to the text data, wherein the first text feature and the second text feature correspond to different semantic levels; Fusing the visual feature corresponding to each pixel in the video frame with the first text feature to determine a first fused feature; Fusing the visual feature corresponding to each pixel in the video frame with the second text feature to determine a second fused feature; Determining, based on the video feature, the first fusion feature, and the second fusion feature, an identification feature for identifying a target to be identified in the video data; and The target recognition module is configured to: recognize the target to be recognized in the video frame based on the recognition feature.

14. An electronic device comprising: one or more processors; and One or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the method of any one of claims 1-12 is executed.

15. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 12.

Images