Data processing method, device, equipment, readable storage medium and program product

Abstract

The embodiment of the present application proposes a data processing method, device, equipment, readable storage medium and program product, which can be applied to cloud technology, artificial intelligence, smart platform, application software, vehicle-mounted, text video retrieval and other fields or scenarios. The method includes: obtaining a description text of the video to be detected, inputting the description text of the video to be detected into a text encoder for processing, and obtaining the text features of the description text of the video to be detected; inputting the video image of the video to be detected into a first video encoder for processing, and obtaining the video features of the video to be detected; according to the text features and the video features, determining the matching results of the description text of the video to be detected and the video to be detected. Through the embodiment of the present application, the generation efficiency and accuracy of the matching results of the video description text and the video to be detected can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] There are a large number of videos stored on the Internet. Based on different business situations, we hope to use a video description text to obtain videos related to the video description text from a large number of videos. At present, manual detection methods are often used to obtain videos that match video description texts. Specifically, when a video description text is given, multiple videos related to the video description text are determined from the video database by manual retrieval, and then the multiple videos are sorted by relevance, and finally the matching results of the video description text are output. The above method relies on manual judgment and analysis of the video, which has strong subjective factors and low efficiency. In addition, manual observation mainly observes the global features of the video, and the perception ability is limited. It is impossible to analyze fine-grained image features, resulting in low accuracy. Therefore, how to improve the generation efficiency and accuracy of the matching results between the video description text and the video to be detected is a problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a data processing method, apparatus, device, readable storage medium and program product, which can improve the generation efficiency and accuracy of the matching results between the video description text and the video to be detected.

[0004] In a first aspect, the present application provides a data processing method, the method comprising:

[0005] Obtain a description text of the video to be detected, input the description text of the video to be detected into a text encoder for processing, and obtain text features of the description text of the video to be detected;

[0006] Inputting the video image of the video to be detected into the first video encoder for processing to obtain the video features of the video to be detected;

[0007] Determine a matching result between the description text of the video to be detected and the video to be detected according to the text features and the video features;

[0008] Among them, the above-mentioned text encoder and the above-mentioned first video encoder are jointly trained in combination with the second video encoder; during the joint training process, the above-mentioned text encoder is used to obtain reference text features of the video description text of the sample video; the above-mentioned second video encoder is used to obtain the first region feature set of the sampling image sequence of the above-mentioned sample video; the above-mentioned first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the above-mentioned sampling image sequence; the above-mentioned global features and the above-mentioned reference text features are used to determine the global loss, the above-mentioned first region feature set and the above-mentioned second region feature set are used to determine the local loss, the network parameters of the above-mentioned text encoder are adjusted according to the above-mentioned global loss, and the network parameters of the above-mentioned first video encoder are adjusted according to the above-mentioned local loss.

[0009] In a second aspect, the present application provides a data processing device, the device comprising:

[0010] An acquisition module is used to acquire a description text of a video to be detected, input the description text of the video to be detected into a text encoder for processing, and obtain text features of the description text of the video to be detected;

[0011] A processing module, used for inputting the video image of the video to be detected into the first video encoder for processing to obtain the video features of the video to be detected;

[0012] The matching module is used to determine the matching result between the description text of the video to be detected and the video to be detected according to the text features and the video features.

[0013] Among them, the above-mentioned text encoder and the above-mentioned first video encoder are jointly trained in combination with the second video encoder; during the joint training process, the above-mentioned text encoder is used to obtain reference text features of the video description text of the sample video; the above-mentioned second video encoder is used to obtain the first region feature set of the sampling image sequence of the above-mentioned sample video; the above-mentioned first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the above-mentioned sampling image sequence; the above-mentioned global features and the above-mentioned reference text features are used to determine the global loss, the above-mentioned first region feature set and the above-mentioned second region feature set are used to determine the local loss, the network parameters of the above-mentioned text encoder are adjusted according to the above-mentioned global loss, and the network parameters of the above-mentioned first video encoder are adjusted according to the above-mentioned local loss.

[0014] In a third aspect, the present application provides a computer device, comprising: a processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are interconnected, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the data processing method.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the data processing method as described above.

[0016] In a fifth aspect, the present application provides a computer program product, which includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the above-mentioned data processing method.

[0017] The present application obtains text features of a description text of a video to be detected through a text encoder, and obtains video features of a video to be detected through a first video encoder; then, based on the text features and the video features, determines the matching results of the description text of the video to be detected and the video to be detected, and automatically obtains the text features and the video features through the structure of the dual encoders and performs feature matching, thereby improving the generation efficiency of the matching results of the video description text and the video to be detected compared to the manual retrieval method.

[0018] The above method uses a method of restoring mask images for model pre-training. Specifically, in the process of jointly training the text encoder and the first video encoder in combination with the second video encoder, the second video encoder is used to extract a first region feature set of a sampled image sequence of the sample video, and the first video encoder is used to extract a second region feature set of a mask image sequence corresponding to the sampled image sequence, and the local loss is determined based on the first region feature set and the second region feature set.

[0019] The network parameters of the first video encoder can be optimized through local loss, so that the optimized first video encoder can use the features extracted by the second video encoder as the reconstruction target and construct the predicted features of the masked area. By minimizing the difference between the predicted features of the first video encoding and the reconstruction target, fine-grained supervision can be applied to the first video encoder to improve the ability of the first video encoder to extract fine-grained features.

[0020] The text encoder is used to extract reference text features of the video description text, the first video encoder is used to extract global features of the mask image sequence, and a global loss is determined based on the reference text features and the global features. The network parameters of the text encoder can be optimized through the global loss to improve the ability of the text encoder to obtain text features that are aligned with the video features extracted by the first video encoder.

[0021] The network parameters of the text encoder are optimized based on the global loss, and the network parameters of the first video encoder are optimized based on the local loss, so that the processing effects of the text encoder and the first video encoder can be improved. The model obtained by this training method is used to perform matching operations between the video description text and the video to be detected, so as to improve the accuracy of the matching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application;

[0024] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0025] Figure 3 is a processing flow diagram of a data processing system provided by an exemplary embodiment of the present application;

[0026] Figure 4 is a flowchart of another data processing method provided by an exemplary embodiment of the present application;

[0027] Figure 5 It is a schematic diagram of a pre-training process of a data processing system provided by an exemplary embodiment of the present application;

[0028] Figure 6 is a schematic block diagram of a data processing device provided by an exemplary embodiment of the present application;

[0029] Figure 7 It is a schematic block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] It should be noted that the descriptions of "first", "second", etc. involved in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0032] The embodiments of the present invention can be applied to fields or scenarios such as cloud computing, cloud Internet of Things, artificial intelligence, smart platforms, application software, vehicle-mounted, text and video retrieval, etc. Several typical application fields or scenarios are introduced below.

[0033] Cloud computing refers to the delivery and use mode of IT infrastructure, which means obtaining the required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use mode of services, which means obtaining the required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model in terms of concept. This application can store data such as video description text and matching results of the video to be detected on the cloud server. When the above different data need to be used, it can be directly obtained on the cloud server, which greatly improves the data acquisition speed.

[0034] Cloud IOT aims to connect the information sensed and instructions received by sensor devices in traditional IoT to the Internet, truly realize networking, and achieve massive data storage and computing through cloud computing technology. Due to the characteristics of IoT, objects are connected to each other and the current operating status of each "object" is perceived in real time. A large amount of data information will be generated in this process. How to aggregate this information and how to filter out useful information from the massive information to support decision-making for subsequent development have become key issues affecting the development of IoT. IoT cloud based on cloud computing and cloud storage technology has therefore become a strong support for IoT technology and applications.

[0035] Intelligent Traffic System (ITS), also known as Intelligent Transportation System, is an effective and comprehensive application of advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) to transportation, service control and vehicle manufacturing, strengthening the connection between vehicles, roads and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy. This application is applied to the intelligent transportation system, and can obtain videos that match the video description text from the video database according to the video description text, which is convenient for retrieval of traffic recording videos for different types of violations, thereby improving the intelligence of the intelligent transportation system and the efficiency of traffic supervision.

[0036] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as IVICS, is a development direction of Intelligent Transportation Systems (ITS). IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0037] This application can also be applied in the field of artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system of simulating, extending and expanding human intelligence, perceiving the environment, acquiring knowledge and using knowledge to obtain the best results using digital computers or machines controlled by digital computers. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, machine learning / deep learning, etc. The solution provided in the embodiment of this application involves machine learning and computer vision technology under artificial intelligence technology, which will be described below.

[0038] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. Machine learning specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies. This application mainly relates to inductive learning technology in machine learning technology. Inductive learning technology aims to summarize and extract general judgment rules and patterns from a large amount of empirical data. It is a learning method that derives general rules from special cases. Specifically, the method proposed in this application uses a text encoder, a first video encoder and a second video encoder to process the video description text and the video to be detected to obtain a matching result. The text encoder, the first video encoder and the second video encoder can be integrated into a text video retrieval model, which is generated based on artificial neural networks, inductive learning and other technologies, and is pre-trained based on sample videos and their video description texts to achieve better model processing effects and improve the accuracy of matching results.

[0039] Computer Vision (CV) Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. Specifically, the method proposed in this application uses computer vision technology to extract image features from sampled images and mask images respectively, and then calculates the regional feature set based on the extracted image features to determine the local loss.

[0040] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, 3D games, etc. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0041] There are a large number of videos stored on the Internet. Based on different business situations, we hope to use a text description to obtain videos related to the text description from a large number of videos. At present, manual detection methods are often used to obtain videos that match the text description. Specifically, when a text description is given, multiple videos related to the text description are determined from the video set to be detected through manual retrieval, and then the multiple videos are sorted by relevance, and finally the video matching results of the text description are output. The above method mainly relies on human eyes to judge and analyze, which has strong subjective factors and low efficiency. In addition, the human eye mainly observes the global features of the video, and its perception ability is limited. It is unable to analyze fine-grained image features, resulting in low accuracy.

[0042] Based on this, this application first proposes a text video retrieval method. When a text description is given, the text video retrieval model first extracts the semantic features of the text description, and then retrieves the video matching the text content from the database by comparing the feature similarity. The pre-training of the text video retrieval model for text video retrieval adopts a "dual encoder" structure to achieve efficient retrieval, in which two separate encoders extract global video features and text features respectively, and optimize the two encoders through comparative learning.

[0043] In the above-mentioned text video retrieval method, this application also provides two model pre-training methods.

[0044] The first method uses two independent encoders to encode text features and video features respectively and project them into the same feature space. This method distinguishes paired and unpaired text-video data through contrastive learning. However, this method ignores the detailed local features of text and video, and also ignores the alignment between text and features.

[0045] The second method realizes cross-modal feature fusion by combining video and text as the input of a joint encoder. This method uses a binary classifier to predict whether the video and text are aligned. However, in this method, every text-video pair must be input into the model during retrieval, resulting in low retrieval efficiency. The coarse-grained alignment of global video and text features by the dual encoder architecture of the above two methods prevents the model from capturing detailed local features, thereby limiting the further improvement of text-video retrieval performance.

[0046] This application introduces Masked Vision Modeling (MVM) into the pre-training of the model. By randomly masking (i.e. covering) a certain proportion of image blocks in the image, the model is trained to reconstruct these masked image blocks, thereby promoting the model's learning of local visual features. Based on the above method, this application proposes a method that combines a dual encoder structure and visual mask modeling for text video retrieval, which achieves fine-grained alignment of video features and text features while ensuring high retrieval efficiency.

[0047] Specifically, the method proposed in the present application randomly masks a certain proportion of video blocks in the video, and the training model reconstructs the masked video blocks that are aligned with the text semantics by reasoning about the visible video blocks in spatial and temporal dimensions. In order to construct a masked video reconstruction target with injected text semantics, a second video encoder (such as a snapshot video encoder) is used to generate a reconstruction target for the masked video block, and the parameters of the first video encoder (such as a video encoder) are used to update the parameters of the second video encoder. In this way, the second encoder aggregates the knowledge of the first video encoder in training, so that the reconstruction target it obtains is gradually aligned with the text semantics.

[0048] By obtaining the reconstruction target through the second video encoder to perform visual mask modeling, the first video encoder in training can be iteratively optimized to capture video features that are aligned with local text features, thereby further enhancing the second video encoder. The above method successfully uses visual mask modeling for video text pre-training without the need for an additional pre-training stage to provide the reconstruction target.

[0049] Specifically, based on the "dual encoder" architecture, an additional second video encoder is used to provide the reconstruction target for visual mask modeling, and this second video encoder is only used for pre-training, which ensures the high efficiency of the dual encoder in retrieval. In each training iteration, by randomly masking a high proportion of video blocks in the spatial and temporal dimensions, the masked mask video is sent to the first video encoder to perform the denoising and auto-encoding process, while the original video is sent to the second video encoder to obtain the reconstruction target. Intuitively, given a high proportion of masked videos, the first video encoder is trained to infer and restore the masked mask video through the reconstruction target, which not only enhances the discriminability of local visual features, but also improves the alignment of fine-grained videos with video description text.

[0050] To sum up, the above method has the following three advantages.

[0051] The first point is to apply visual mask modeling to video-text pre-training by using a dual encoder structure, and to give full play to the advantages of the dual encoder structure combined with visual mask modeling in fine-grained video feature understanding and local semantic alignment of video text.

[0052] The second point is that the second video encoder is used to provide a reconstruction target for visual mask modeling, and the first video encoder iteratively optimized by visual mask modeling can inversely enhance the second video encoder.

[0053] The third point is that the above method was tested on multiple text-to-video retrieval datasets and achieved better results than existing methods. This application also further evaluated zero-sample action recognition (a task that can be regarded as video-to-text retrieval), and the method proposed in this application also achieved better results.

[0054] It can be understood that in the specific implementation of the present application, related data such as the video to be detected and the video description text are involved. When the above embodiments of the present application are applied to specific products or technologies, the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0055] This application will be specifically described by the following examples:

[0056] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the architecture of a data processing system provided by an exemplary embodiment of the present application. Figure 1 As shown, the data processing system may specifically include a terminal device 101 and a server 102. The terminal device 101 and the server 102 are connected via a network, such as a local area network, a wide area network, a mobile Internet, etc. The operation object performs operations such as data input on the browser or client application of the terminal device 101. The server 102 may provide various data processing services for the operation object in response to the operation.

[0057] Specifically, the server 102 can obtain a description text of the video to be detected (the obtained description text of the video to be detected can be data obtained by the terminal device 101 through the data input operation of the operating object); the server 102 inputs the description text of the video to be detected into a text encoder for processing to obtain text features of the description text of the video to be detected, and inputs a video image of the video to be detected (wherein, the video to be detected can be specified by the operating object when performing a data input operation on the terminal device 101, or it can be a video database existing in the server 102) into the first video encoder for processing to obtain video features of the video to be detected; the server 102 determines the matching result of the description text of the video to be detected and the video to be detected based on the text features and the video features; the server 102 finally sends the matching result to the terminal device 101, and the terminal device 101 displays the matching result.

[0058] The terminal device 101 is also referred to as a terminal, user equipment (UE), access terminal, user unit, mobile device, user terminal, wireless communication device, user agent or user device. The terminal device may be a smart home appliance, a handheld device with wireless communication function (such as a smart phone, a tablet computer), a computing device (such as a personal computer (PC), a vehicle terminal, an intelligent voice interaction device, a wearable device or other smart device, but is not limited thereto.

[0059] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0060] It is understandable that the system architecture diagram described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. For example, the data processing method provided in the embodiment of the present application can be executed by the server 102, and can also be executed by other servers or server clusters that are different from the server 102 and can communicate with the terminal device 101 and / or the server 102. It is known to those skilled in the art that Figure 1The number of terminal devices and servers in is only illustrative. According to the business implementation needs, any number of terminal devices and servers can be configured. In addition, with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems. In subsequent embodiments, the terminal device will refer to the above-mentioned terminal device 101, and the server will refer to the above-mentioned server 102, which will not be repeated in subsequent embodiments.

[0061] See also Figure 2 , Figure 2 is a flow chart of a data processing method provided by an exemplary embodiment of the present application, wherein the method is applied to a text video retrieval system (the text video retrieval system can be configured in Figure 1 Taking the server 102 or the terminal device 101 as an example, the method may include the following steps:

[0062] S201, obtaining a description text of a video to be detected, inputting the description text of the video to be detected into a text encoder for processing, and obtaining text features of the description text of the video to be detected.

[0063] In the embodiment of the present application, the video description text to be detected is the target text for the text video retrieval task (that is, determining the matching result between the video description text to be detected and the video to be detected). The text encoder can extract the text features of the text, and the video description text to be detected is processed by the text encoder to obtain the text features of the video description text to be detected. Among them, the text features of the video description text to be detected are used to calculate the similarity with the video features of the video to be detected determined in the subsequent steps, so as to determine the matching result between the video description text to be detected and the video to be detected.

[0064] S202: Input the video image of the video to be detected into a first video encoder for processing to obtain video features of the video to be detected.

[0065] In the embodiment of the present application, the video to be detected is a target video for a text video retrieval task. The first video encoder can extract image features of the image, and the first video encoder processes the video image of the video to be detected to obtain video features that can characterize the video to be detected.

[0066] In one embodiment, the video image of the video to be detected is extracted from the video to be detected, and the video image of the video to be detected can be all the image frames obtained after extracting the video to be detected frame by frame, or can be a part of the image frames obtained after sampling the video to be detected according to a certain sampling rule (such as interval sampling, etc.). Since the video usually presents similar visual content at the same position of adjacent frames (or close frames), the image features of multiple image frames obtained by the frame-by-frame sampling method are similar, resulting in a low accuracy of the matching result obtained using the image data. By setting the sampling rule for sampling, the amount of sampled image data can be reduced under the premise of ensuring the subsequent processing effect, the sampling efficiency of the video to be detected and the efficiency of processing the sampled image are improved, and the accuracy of the matching result obtained using the image data is improved.

[0067] In one embodiment, interval sampling (interval sampling is considered to be the need to set an interval frame number for acquisition when acquiring multiple video images, and the set interval frame number can be large or small, and is not limited to its maximum threshold or minimum threshold) can be used to obtain the video image of the video to be detected, and the set interval frame number can be a fixed value or a non-fixed value. Specifically, the text video retrieval system can perform interval sampling by setting a set interval frame number for acquiring image frames (for example, setting the interval frame number to 1 frame, that is, sampling is performed after every interval of 1 frame of image, at this time, it can be ensured that any two acquired video images are not adjacent frame images; for example, setting the interval frame number to 100 frames, that is, sampling is performed after every interval of 100 frames of image). According to this idea, all image frames of the video to be detected are processed, and the acquired image frames are used as video image inputs of the video to be detected.

[0068] In one embodiment, in the same video to be detected, the larger the interval between any two video images, the smaller the correlation between the images. In order to increase the difficulty of mask image reasoning to achieve a better model processing effect, a special case of interval sampling, i.e., sparse sampling (sparse sampling is considered as two video images collected continuously need to reach a larger interval threshold, i.e., a minimum threshold is defined. The interval threshold should be determined according to the specific business situation) can be used to obtain the video image of the video to be detected. Specifically, the text video retrieval system can perform sparse sampling by setting an interval threshold for collecting image frames (for example, if the interval threshold is 15 frames, then sampling can be performed after every 20 frames of images); or by setting a random interval range for collecting image frames (for example, if the interval threshold is 15 frames, then a target interval value can be determined within the range of [20,40], that is, after collecting one frame, a target interval value (for example, 25) is randomly determined from [20,40], and then image frames after the target interval value from the previous frame (that is, image frames with an interval of 25 frames from the previous frame) are collected for sparse sampling. According to this idea, all image frames of the video to be detected are processed, and the collected image frames are used as video image inputs of the video to be detected).

[0069] S203. Determine the matching result between the video description text to be detected and the video to be detected based on the text features and the video features; wherein the text encoder and the first video encoder are obtained by joint training in combination with the second video encoder; during the joint training process, the text encoder is used to obtain the reference text features of the video description text of the sample video; the second video encoder is used to obtain the first region feature set of the sampled image sequence of the sample video; the first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the sampled image sequence; the global features and the reference text features are used to determine the global loss, the first region feature set and the second region feature set are used to determine the local loss, the network parameters of the text encoder are adjusted according to the global loss, and the network parameters of the first video encoder are adjusted according to the local loss.

[0070] In the embodiment of the present application, the text feature can represent the feature information of the video description text to be detected, and the video feature can represent the feature information of the video to be detected. By calculating the similarity of the text feature and the video feature, the text video retrieval system can determine the matching result of the video description text to be detected and the video to be detected. The matching result can be presented in the form of a matching score (for example, the higher the matching degree, the higher the matching score; the lower the matching degree, the lower the matching score); the matching result can also be presented in the form of a binary classification (for example, the binary classification result includes a match and a mismatch).

[0071] In one embodiment, the text video retrieval system can determine the similarity between the text features and the video features by performing a dot product calculation on the text features and the video features. Generally speaking, the higher the similarity, the higher the matching degree between the video description text to be detected and the video to be detected; the lower the similarity, the lower the matching degree between the video description text to be detected and the video to be detected. By using an independent dual encoder structure to obtain text features and video features, and calculating the similarity by dot product to determine the matching result, the generation efficiency of the matching result between the video description text and the video to be detected is improved.

[0072] In an embodiment of the present application, a text encoder and a first video encoder are jointly trained in combination with a second video encoder; during the joint training process, the text encoder is used to obtain reference text features of a video description text of a sample video; the second video encoder is used to obtain a first region feature set of a sampled image sequence of a sample video; the first video encoder is used to obtain global features and a second region feature set of a mask image sequence of a sampled image sequence; the global features and reference text features are used to determine a global loss, and the first region feature set and the second region feature set are used to determine a local loss, the network parameters of the text encoder are adjusted based on the global loss, and the network parameters of the first video encoder are adjusted based on the local loss.

[0073] In one embodiment, the network parameters of the text encoder may also be obtained by jointly adjusting the global loss and the local loss; the network parameters of the first video encoder may also be obtained by jointly adjusting the global loss and the local loss.

[0074] Specifically, a set of training data may include sample videos (including a sampled image sequence and a masked image sequence after masking the sampled image sequence) and video description text. The reference text features of the video description text obtained by the text encoder may represent feature information of the video description text; the second region feature set of the masked image sequence obtained by the first video encoder may represent local feature information of the masked image sequence; and the first region feature set of the sampled image sequence obtained by the second video encoder may represent local feature information of the sampled image sequence.

[0075] The local loss is determined by the first region feature set and the second region feature set (the role of the second video encoder can be understood as providing a reconstruction target for visual mask modeling by generating the first region feature set; the role of the first video encoder can be understood as constructing a masked region by generating the second region feature set). By minimizing the distance between each set of feature pairs that match each other in the predicted first region feature set and the second region feature set, fine-grained supervision is applied to the first video encoder to improve the processing effect of the first video encoder, thereby improving the accuracy of the matching results between the video description text and the video to be detected.

[0076] In one embodiment, the text video retrieval system can use a contrastive learning method to maximize the similarity between the video features of the video to be detected and the text features of the matching video description text, and minimize the similarity between the video features of the video to be detected and the text features of the unmatched video description text, thereby improving the processing effect of the first video encoder and the text encoder.

[0077] Among them, the specific process of jointly training the above-mentioned text encoder, the first video encoder and the second video encoder will be introduced in subsequent embodiments and will not be repeated in this embodiment.

[0078] In one embodiment, the video to be detected is included in a video database, and the video database includes multiple videos. After the above-mentioned determination of the matching result between the description text of the video to be detected and the video to be detected, the following steps may also be included: according to the matching result between each video in the multiple videos and the description text of the video to be detected, a matching video that matches the description text of the video to be detected is determined from the video database.

[0079] Wherein, the video database includes multiple videos, and each of the multiple videos can determine the corresponding matching result according to the steps S201-S203; then determine the matching video that matches the description text of the video to be detected from the multiple matches corresponding to the multiple videos. For example, when the matching result is presented in the form of a matching score, the text video retrieval system can use the video corresponding to the matching result that meets the matching score threshold as the matching video that matches the description text of the video to be detected; when the matching result is presented in the form of a binary classification, the text video retrieval system can use the video corresponding to the matching result with a matching result of "match" as the matching video that matches the description text of the video to be detected).

[0080] See also Figure 3 , Figure 3 This is a processing flow diagram of a data processing system (i.e., a text video retrieval system) provided by an exemplary embodiment of the present application. The figure includes a text encoder and a first video encoder. The text video retrieval system inputs the description text of the video to be detected (e.g., "two small boats are floating in the lake in front of the mountain") into the text encoder to obtain the text features of the description text of the video to be detected; the text video retrieval system simultaneously inputs the video image of the video to be detected (the video image can be obtained by the text video retrieval system after sampling the video to be detected, or it can be a video image directly obtained by the text video retrieval system) into the first video encoder to obtain the video features of the video to be detected; the text video retrieval system compares the similarity of the text features and the video features to obtain the matching result of the description text of the video to be detected and the video to be detected.

[0081] The beneficial effect of the present application is that: the present application obtains text features of a video description text to be detected through a text encoder, and obtains video features of a video to be detected through a first video encoder; then, based on the text features and the video features, similarities are determined by calculating the dot product of the text features and the video features, and then the matching results of the video description text to be detected and the video to be detected are determined. The text features and video features are automatically obtained and feature matching is performed through the structure of the above-mentioned dual encoders, which improves the generation efficiency of the matching results of the video description text and the video to be detected compared to the manual retrieval method.

[0082] The above method uses a method of restoring visual masks for model pre-training. Specifically, in the process of jointly training the text encoder and the first video encoder in combination with the second video encoder, the reference text features obtained by the text encoder can represent the feature information of the video description text; the global features and the second regional feature set obtained by the first video encoder can respectively represent the global feature information and local feature information of the mask image sequence; the first regional feature set obtained by the second video encoder can represent the local features of the sampled image sequence. The first regional feature set of the sampled image sequence of the sample video is extracted by the second video encoder, and the second regional feature set of the mask image sequence corresponding to the sampled image sequence is extracted by the first video encoder, and the local loss is determined based on the first regional feature set and the second regional feature set.

[0083] The network parameters of the first video encoder can be optimized through local loss, so that the optimized first video encoder can use the features extracted by the second video encoder as the reconstruction target and construct the predicted features of the masked area. By minimizing the difference between the predicted features of the first video encoding and the reconstruction target, fine-grained supervision can be applied to the first video encoder to improve the ability of the first video encoder to obtain fine-grained features.

[0084] The text encoder is used to extract reference text features of the video description text, the first video encoder is used to extract global features of the mask image sequence, and a global loss is determined based on the reference text features and the global features. The network parameters of the text encoder can be optimized through the global loss to improve the ability of the text encoder to obtain text features that are aligned with the video features extracted by the first video encoder.

[0085] The network parameters of the text encoder are optimized based on the global loss, and the network parameters of the first video encoder are optimized based on the local loss, so that the processing effects of the text encoder and the first video encoder can be improved. The model obtained by this training method is used to perform matching operations between the video description text and the video to be detected, so as to improve the accuracy of the matching results.

[0086] The present application can maximize the similarity between the video features of the video to be detected and the text features of the matching video description text by a comparative learning method, and minimize the similarity between the video features of the video to be detected and the text features of the unmatched video description text, thereby improving the processing effect of the first video encoder and the text encoder. The model obtained by this training method is used to perform the matching operation of the video description text and the video to be detected, which can improve the accuracy of the matching result.

[0087] The present application can be applied to text video retrieval tasks, by calculating the matching results of each video in the video database and the description text of the video to be detected, and then determining the matching video that matches the description text of the video to be detected based on multiple matching results, and outputting the results. The present application also proposes to set sampling rules (such as interval sampling, etc.) for sampling, thereby generating a sampled image sequence, which can reduce the amount of sampled image data while ensuring the effect of subsequent processing, improve the sampling efficiency of the video to be detected and the efficiency of processing the sampled images, and improve the accuracy of the matching results obtained using the image data.

[0088] See also Figure 4 , Figure 4 is a flow chart of a data processing method provided by an exemplary embodiment of the present application, wherein the method is applied to a text video retrieval system (the text video retrieval system can be configured in Figure 1 Taking the server 102 or the terminal device 101 in the example as an example, the method may include the following steps:

[0089] S401: Obtain video description text and a sampled image sequence of a sample video, and obtain a mask image sequence corresponding to the sampled image sequence.

[0090] In an embodiment of the present application, a set of training data may include a sample video (including a sampled image sequence and a masked image sequence after masking the sampled image sequence) and a video description text, which matches the display content of the sample video. The sampled image sequence is extracted from the sample video and serves as input data of the second video encoder. The masked image sequence is obtained by processing the sampled image sequence and serves as input data of the first video encoder.

[0091] In one embodiment, the process of obtaining the sample image sequence of the sample video can be implemented according to the following steps: performing interval sampling processing on the sample video to obtain a sample image sequence, and any two sample images in the sample image sequence are not adjacent.

[0092] The sampled image sequence carries an image sequence, which means that each frame of the sampled image is arranged in the order in which it appears in the sample video. For example, the text video retrieval system can sample the second, fifth, and eighth frames of the sample video, respectively, to generate a sampled image sequence. Then the sequence of the second frame is 1, the sequence of the fifth frame is 2, and the sequence of the eighth frame is 3.

[0093] In one embodiment, the text video retrieval system can use a sample video and a video description text of the sample video to generate multiple sets of training data (i.e., use a sample video and a video description text of the sample video for multiple trainings), and obtain multiple sets of sample image sequences corresponding to a sample video by setting different acquisition methods in the process of acquiring the sample image sequence; and generate multiple mask image sequences corresponding to a set of sample image sequences by masking different image areas in the process of masking the sample image sequence.

[0094] Exemplarily, the sample video includes 10 frames. When collecting the first set of training data, 1, 4, and 7 frames are collected as a group, 2, 5, and 8 frames are collected as a group, and 3, 6, and 9 frames are collected as a group; three sets of sampling image sequences are generated based on the three sets of images obtained, and then the three sets of sampling image sequences are masked to obtain three sets of masked image sequences, thus obtaining three sets of training data. In the masking process, for each set of sampling image sequences, multiple masked image sequences can be generated by masking different image regions (for example, masking the upper left corner region, the lower left corner region, the upper right corner region, and the lower right corner region to generate four sets of masked image sequences).

[0095] Through the above method, a large amount of training data can be generated from a small amount of sample videos and their video description texts for pre-training of the model, reducing the amount of data and improving training efficiency. It should be noted that the above selection of the number of frames collected for each set of training data and the masked area is only exemplary, and multiple sets of training data can also be obtained by randomly generating the number of frames collected for each set of training data and the masked area. The specific method for generating training data should be selected according to the actual business situation, and this application does not limit it.

[0096] In one embodiment, the text video retrieval system can set an interval threshold for collecting image frames (for example, 20, that is, sampling is performed after every 20 frames of images); the text video retrieval system can also set a random interval range for collecting image frames (for example, [20,40], that is, after collecting one frame, randomly determine a target interval value (for example, 25) from [20,40], and then collect image frames after the target interval value from the previous frame (that is, image frames with an interval of 25 frames from the previous frame), process all image frames of the video to be detected according to this idea, and use the collected image frames as video image inputs of the video to be detected). Through the above method, it can be ensured that the image features of multiple frames in the sampled image sequence have certain differences, thereby improving the effect of image feature extraction.

[0097] In one embodiment, the above-mentioned acquisition of the mask image sequence corresponding to the sampling image sequence can be implemented according to the following steps: mask processing is performed on each sampling image in the sampling image sequence to obtain a mask image sequence, and the masked image areas in each sampling image are matched.

[0098] Since videos usually present similar visual content in adjacent frames (adjacent blocks in time and space) or at the same position in nearby frames, the image features of multiple image frames obtained by the text video retrieval system through the frame-by-frame sampling method are similar. The covered mask area can be easily restored by fast interpolation of adjacent blocks in time and space, resulting in lower accuracy of the matching results obtained using the image data.

[0099] In order to make the task of visual mask modeling more challenging and to improve the data processing model's understanding of the spatial and temporal dimensions, the present application adopts a tubular masking strategy to mask the sample video along the spatial and temporal dimensions (that is, to mask the same position of each sampled image in the sampled image sequence obtained after interval sampling, rather than independently masking the content of each sampled image). The above method suppresses the first video encoder from reconstructing the masked video content (i.e., the masked area) by interpolating the visual content of adjacent frames. Instead, it is necessary to perform visual reasoning on the visible video content in the spatial and temporal dimensions, thereby improving the processing effect of the model and the accuracy of the matching results. Moreover, through the interval sampling method, the amount of sampled image data can be reduced while ensuring the processing effect, thereby improving the sampling efficiency of the video to be detected and the processing efficiency of the sampled images.

[0100] S402: Input the video description text of the sample video into an initial text encoder for feature extraction to obtain reference text features.

[0101] The specific implementation of step S402 refers to the related description of step S201 in the above embodiment, which will not be repeated in this embodiment.

[0102] S403, inputting the sampled image sequence into the initial second video encoder for feature extraction to obtain a first region feature set, and inputting the masked image sequence into the initial first video encoder for feature extraction to obtain global features and a second region feature set.

[0103] In the embodiment of the present application, the initial first video encoder and the initial second video encoder are initialization models before pre-training. The second video encoder (and the initial second video encoder) can perform feature extraction processing on the image to obtain feature information of the image. The second video encoder has the same structure as the first video encoder and is used to process different types of images (i.e., sampled image sequences and masked image sequences) in the present application.

[0104] In one embodiment, the process of inputting the mask image sequence into the initial first video encoder for feature extraction to obtain the second region feature set can be implemented according to the following steps:

[0105] (1) Inputting a mask image sequence into an initial first video encoder for processing, dividing the target mask image in the mask image sequence into regions, and extracting regional features of each divided image region; wherein the regional features of a first image region including a mask region in the target mask image are predicted features obtained based on a non-masked portion in a reference mask image and a non-masked portion in the target mask image, and the predicted features are close to features extracted by the initial second video encoder from a second image region matching the first image region in a sampled image sequence; the target mask image is any mask image in the mask image sequence, and the reference mask image is a mask image in the mask image sequence other than the target mask image.

[0106] In an embodiment of the present application, the target mask image is any mask image in a mask image sequence, and the text video retrieval system performs region division processing on the target mask image and extracts the regional features of each divided image region. The regional features of the first image region are predicted features predicted based on the non-masked portion in the reference mask image and the non-masked portion in the target mask image, and the reference mask image is a mask image other than the target mask image in the mask image sequence. In other words, the text video retrieval system obtains the mask region of the target mask image by inferring the non-masked region of the target mask image and the non-masked region of the reference mask image, and by performing visual reasoning on the visible video content in the spatial dimension and the temporal dimension, the data processing model's understanding of the spatial dimension and the temporal dimension is enhanced, thereby improving the processing effect of the model and also improving the accuracy of the matching results.

[0107] In one embodiment, when inferring the mask region of the target mask image, the text video retrieval system can perform inference based on multiple reference mask images, thereby improving the accuracy of mask region inference. At the same time, multiple reference mask images can also be weighted, for example, the weight of the reference mask image closer to the target mask image in the mask image sequence is set to be larger; the weight of the reference mask image farther from the target mask image in the mask image sequence is set to be smaller; or when the distance threshold is met, the reference mask image farther away is ignored, thereby selectively selecting data for performing mask region inference, further improving the accuracy of mask region inference.

[0108] In one embodiment, the first image region is the region that needs to be restored, and the first image region can be a mask region, or a region including the mask region (that is, in addition to the mask region, it also includes a portion of the region that is not masked), or the entire sampled image. The above method improves the flexibility of data selection and region division in this application.

[0109] (2) A second region feature set is obtained based on the region features extracted from each mask image in the mask image sequence.

[0110] In an embodiment of the present application, the mask image sequence includes multiple mask images. For each mask image, the text video retrieval system can use the method (1) in step S403 to obtain multiple regional features of each mask image, and then merge the multiple regional features of the multiple mask images according to the time dimension and the space dimension as the second regional feature set.

[0111] Exemplarily, the mask image sequence includes M mask images, and the text video retrieval system performs region division processing on each mask image to obtain N divided image regions, and extracts N regional features corresponding to the N divided image regions, then the second regional feature set contains M*N regional features. The individual regional features in the second regional feature set are arranged according to the sequence of the individual mask images in the mask image sequence, and the position of each regional feature in the mask image. Similarly, the sampling image sequence uses the same method for region division and arrangement, which will not be repeated in this embodiment.

[0112] In one embodiment, the process of inputting the mask image sequence into the initial first video encoder to extract features and obtain global features can be implemented by the following methods.

[0113] The first method is to input the M mask images included in the mask image sequence into the first video encoder for processing to obtain M global features to be fused corresponding to the M mask images; then perform feature fusion processing on the M global features to be fused to obtain the global features of the mask image sequence.

[0114] The second way is to perform image merging processing on the M mask images included in the mask image sequence to obtain a merged mask image, and then input the merged mask image into the first video encoder for processing to obtain the global features of the mask image sequence.

[0115] The third method is to input the mask image sequence into the initial first video encoder for feature extraction to obtain a second region feature set, and then perform fusion processing on the second region feature set to obtain the global features of the mask image sequence. By obtaining the global features of the mask image sequence through the above multiple methods, the flexibility of data processing can be improved.

[0116] S404: Determine a global loss according to the global features and the reference text features, and determine a local loss according to the first regional feature set and the second regional feature set.

[0117] In the embodiment of the present application, the global feature can characterize the feature information of the sampled image sequence of the sample video; the reference text feature can characterize the feature information of the video description text; the first region feature set can characterize the local features of the sampled image sequence; the second region feature set can characterize the local features of the masked image sequence. The text video retrieval system can determine the local loss through the first region feature set and the second region feature set (the role of the second video encoder can be understood as providing a reconstruction target for visual mask modeling by generating the first region feature set; the role of the first video encoder can be understood as constructing the masked area by generating the second region feature set).

[0118] The text video retrieval system can apply fine-grained supervision to the first video encoder by minimizing the distance between each set of feature pairs that match each other in the predicted first region feature set and the second region feature set (that is, minimizing local losses) to improve the processing effect of the first video encoder, thereby improving the accuracy of the matching results between the video description text and the video to be detected.

[0119] In one embodiment, the first region feature set is composed of region features extracted by the initial second encoder for each sampled image in the sampled image sequence, and each region feature corresponds to a divided image region of the sampled image.

[0120] Among them, the process of the initial second encoder determining the first region feature set based on the sampled image sequence, please refer to the relevant description of inputting the mask image sequence into the initial first video encoder for feature extraction in step S403 to obtain the global features and the second region feature set, which will not be repeated in this embodiment.

[0121] The above-mentioned determination of the global loss according to the global features and the reference text features, and the determination of the local loss according to the first region feature set and the second region feature set can be implemented according to the following steps.

[0122] (1) Matching the global features with the reference text features, and determining the global loss based on the matched global features and the matched reference text features.

[0123] In the embodiment of the present application, the global feature is the feature of the image dimension, and the reference text feature is the feature of the text dimension. In order to align the global feature and the reference text feature in the same feature dimension, the text video retrieval system needs to match the global feature and the reference text feature, thereby improving the accuracy and effectiveness of the global loss.

[0124] In one embodiment, the text video retrieval system can perform multimodal feature alignment processing on global features and reference text features, and determine the global loss based on the global features and reference text features after the multimodal feature alignment processing. The multimodal feature alignment processing can project the global features and the reference text features into the same feature space, and the method can use contrastive learning to distinguish paired and unpaired text video data. In addition, the sample video and the video description text can be connected as the input of a joint encoder to achieve cross-modal feature fusion, and then a binary classifier is used to predict whether the sample video and the video description text are aligned.

[0125] (2) For any regional feature in the first regional feature set, a matching regional feature that matches any regional feature is determined from the second regional feature set.

[0126] In an embodiment of the present application, the matching area feature is determined from the second area feature set, and any of the above area features is determined from the first area feature set, and any of the above area features corresponds to the matching area feature in the spatial dimension and the temporal dimension (that is, the features corresponding to the same sequence and the same image area in the sampling image sequence and the mask image sequence).

[0127] (3) Determine the difference parameter between any regional feature and the matching regional feature, and determine the local loss based on the difference parameter.

[0128] In the embodiment of the present application, for the first region feature set and the second region feature set, multiple difference parameters corresponding to different divided image regions (that is, difference parameters between multiple groups of any region features and matching region features) can be obtained, and each difference parameter can characterize the feature difference between the matched divided image regions in the sampling image sequence and the mask image sequence, and is used to calculate the local loss. Among them, the above-mentioned local loss can refer to a set of multiple local losses corresponding to multiple groups of any region features and matching region features, and can also refer to the local loss of any group of any region features and matching region features.

[0129] S405. Adjust the network parameters of the initial text encoder according to the global loss, and determine the trained text encoder according to the adjusted text encoder; and adjust the network parameters of the initial first video encoder according to the local loss, and determine the trained first video encoder according to the adjusted first video encoder.

[0130] In one embodiment, the above process of adjusting the network parameters of the initial text encoder according to the global loss and adjusting the network parameters of the initial first video encoder according to the local loss can be implemented in the following manner: for a set of sample videos and video description texts, the text video retrieval system adjusts the network parameters of the initial text encoder by the global loss and adjusts the network parameters of the initial first video encoder by the local loss, so that the first video encoder can capture the local visual semantics (i.e., the second regional feature set) aligned with the text features of the video description text. In this process, for multiple local losses corresponding to multiple groups of any regional features and matching regional features, the network parameters of the initial first video encoder are adjusted iteratively using each local loss (the network parameters of the initial text encoder can also be adjusted).

[0131] In one embodiment, the network parameters of the text encoder may also be obtained by jointly adjusting the global loss and the local loss; the network parameters of the first video encoder may also be obtained by jointly adjusting the global loss and the local loss.

[0132] For multiple groups of sample videos and video description texts, the network parameters of the initial text encoder and the initial first video encoder can be iteratively optimized based on the above method, and finally the trained text encoder and the trained first video encoder can be determined based on the adjusted text encoder and the adjusted first video encoder.

[0133] In one embodiment, the following steps may be performed based on the above method.

[0134] (1) After the network parameters of the first video encoder are adjusted, the network parameters of the second video encoder are updated according to the adjusted network parameters of the first video encoder to obtain the adjusted second video encoder.

[0135] (2) In combination with the adjusted second video encoder, the adjusted text encoder and the adjusted first video encoder are jointly trained again to obtain a trained text encoder and a trained first video encoder; wherein the features obtained by the adjusted second video encoder for the sampled image of the sample video are close to the text features obtained by the adjusted text encoder for the video description text of the sample video.

[0136] In an embodiment of the present application, the text encoder, the first video encoder and the second video encoder can be jointly trained. Since the first video encoder and the second video encoder have the same structure, after the network parameters of the second video encoder are updated (for example, parameter synchronization, parameter adjustment, etc.) according to the adjusted network parameters of the first video encoder, the second video encoder can aggregate the knowledge of the first video encoder, so that the features obtained by the second video encoder for the sampled image of the sample video (including global features and local features of the sample video) are close to the text features obtained by the adjusted text encoder for the video description text of the sample video (that is, the reconstruction target of the mask image obtained by the second video encoder is gradually aligned with the reference text features), thereby improving the accuracy of the model processing, thereby obtaining a more accurate matching result between the video description text and the video to be detected.

[0137] In one embodiment, the parameters of the second video encoder can be updated in the following manner: within a training cycle (i.e., using the same set of sample videos and video description texts), the network parameters of the second video encoder are fixed, then in the kth training cycle, the update formula of the network parameters of the second video encoder is as follows.

[0138] {θ s} k =λ{θ s} k-1 +(1-λ){θ v} k-1

[0139] Among them, {θ s} k is the network parameter of the second video encoder at the end of the kth training cycle, {θ s} k-1 is the network parameter of the second video encoder at the end of the k-1th training cycle, {θ v} k-1 is the network parameter of the first video encoder at the end of the k-1th training cycle, and λ is a constant.

[0140] By updating the network parameters of the second video encoder through the above method, the optimization process of the second video encoder is made smoother, so that the reconstruction target of the mask image obtained by the second video encoder is gradually aligned with the reference text features, thereby obtaining a more accurate matching result between the video description text and the video to be detected.

[0141] S406: Obtain a description text of the video to be detected, input the description text of the video to be detected into a text encoder for processing, and obtain text features of the description text of the video to be detected.

[0142] S407: Input the video image of the video to be detected into the first video encoder for processing to obtain the video features of the video to be detected.

[0143] S408: Determine a matching result between the to-be-detected video description text and the to-be-detected video according to the text features and the video features.

[0144] The specific implementation of steps S406 to S408 refers to the related description of steps S201 to S203 in the above embodiment, which will not be repeated in this embodiment.

[0145] See also Figure 5 , Figure 5 This is a schematic diagram of the pre-training process of a data processing system (i.e., a text video retrieval system) provided by an exemplary embodiment of the present application. During the pre-training process, in addition to the text encoder and the first video encoder, a second video encoder is also included. The text video retrieval system inputs the video description text of the sample video (for example, "two boats are floating in the lake in front of the mountain") into the text encoder to obtain reference text features of the video description text; the text video retrieval system inputs the sample image sequence of the sample video (the sample image sequence can be obtained by the text video retrieval system after sampling the sample video, or it can be a sample image sequence directly obtained by the text video retrieval system) into the second video encoder to obtain a first regional feature set of the sample image sequence.

[0146] The text video retrieval system simultaneously inputs the mask image sequence of the sample video (the mask image sequence can be obtained by the text video retrieval system after masking the sampling image sequence of the sample video, or can be the mask image sequence directly obtained by the text video retrieval system) into the first video encoder to obtain the second regional feature set of the mask image sequence and the global features of the mask image sequence.

[0147] The local loss is determined according to the first region feature set and the second region feature set (the second region feature set is supervised by the first region feature set at the same time), and the global loss is determined according to the reference text features and the global features (that is, the similarity of the reference text features and the global features is compared); the text video retrieval system then jointly trains the text encoder, the first video encoder and the second video encoder according to the local loss and the global loss, and finally obtains the trained model.

[0148] Based on the above text-video retrieval and pre-training methods, this application conducted relevant tests on four retrieval datasets.

[0149] The test methods include zero-sample and training set fine-tuning (the zero-sample method refers to using the same model parameters for each data set for the test task, that is, the model network parameters are not updated; the training set fine-tuning method refers to updating the model parameters for each data set, and applying the updated model to subsequent test tasks). The specific test results are shown in the following table.

[0150] Table 1a

[0151]

[0152]

[0153] Table 1b

[0154] method Years Visual Input Pre-training dataset quantity R@1 R@5 R@10 MedR ActBERT 2020 ResNet-3D HowTo100M 120M 16.3 42.8 56.9 10.0 UniV 2020 S3D HowTo100M 110M 21.2 49.6 63.1 6.0 MMT 2020 S3D HowTo100M 120M 26.6 57.1 69.6 4.0 HERO 2021 SlowFast TV,HowTo100M 120M 16.8 43.4 57.7 - NoiseEst 2021 ResNext-101 HowTo100M 110M 17.4 41.6 53.6 8.0 ClipBert 2021 Raw Videos COCO,VisGenome 5.6M 22.0 46.8 59.9 6.0 AVLnet 2021 ResNext-101 HowTo100M 120M 27.1 55.6 66.6 4.0 VLM 2021 S3D HowTo100M 120M 28.1 55.5 67.4 4.0 TACo 2021 I3D,S3D HowTo100M 120M 28.4 57.8 71.2 4.0 SupportSet 2021 R(2+1)D-34 HowTo100M 120M 30.1 58.5 69.3 3.0 VideoCLIP 2021 S3D HowTo100M 110M 30.9 55.4 66.8 - Frozen 2021 Raw Videos CC3M,WebVid-2M 5.5M 31.0 59.5 70.5 3.0 Ours 2022 Raw Videos CC3M,WebVid-2M 5.5M 37.7 63.6 73.8 3.0

[0155] Table 1a and Table 1b show the text-to-video retrieval results on the MSR-VTT dataset. The test results of the zero-shot method are shown in Table 1a, and the test results of the training set fine-tuning method are shown in Table 1b. The table includes multiple indicators, including method, year, visual input, pre-trained dataset, number, R@1, R@5, R@10, and MedR. Among them, the "visual input" indicator is used to indicate the feature network that obtains video features. The "Raw Videos" under this indicator means that the original video is directly used for training without pre-extracting video features through the feature network for training; the "number" indicator is used to indicate the number of pre-trained video text pairs; R@K (such as R@1, R@5, and R@10) can be understood as the accuracy of the true result existing in the first K prediction results; MedR can be understood as the Rth true result among multiple prediction results. Higher R@K and lower MedR indicate better results.

[0156] It can be seen from the table that Ours method (i.e., the method proposed in this application) has higher R@K and lower MedR, and achieves better processing results than other methods.

[0157] Table 2a

[0158] method R@1 R@5 R@10 MedR NoiseEst 13.7 35.7 47.7 12.0 SupportSet 21.4 46.2 57.7 6.0 Frozen 33.7 64.7 76.3 3.0 Ours 44.4 76.2 87.0 2.0

[0159] Table 2b

[0160] method R@1 R@5 R@10 MedR NoiseEst 20.3 49.0 63.3 6.0 SupportSet 28.4 60.0 72.9 4.0 Frozen 45.6 79.8 88.2 2.0 Ours 53.9 83.5 90.2 1.0

[0161] Table 2a and Table 2b show the text-to-video retrieval results on the MSVD dataset. The test results of the zero-shot method are shown in Table 2a, and the test results of the training set fine-tuning method are shown in Table 2b. The table includes multiple indicators, including R@1, R@5, R@10, and MedR. Among them, R@K (such as R@1, R@5, and R@10) can be understood as the accuracy of the true result existing in the first K predicted results; MedR can be understood as the Rth true result among multiple predicted results. Higher R@K and lower MedR indicate better results.

[0162] It can be seen from the table that Ours method (i.e., the method proposed in this application) has higher R@K and lower MedR, and achieves better processing results than other methods.

[0163] Table 3a

[0164] method R@1 R@5 R@10 MedR AVLnet 1.4 5.9 9.4 273.5 NoiseEst 4.2 11.6 17.1 119.0 Frozen 9.3 22.0 30.1 51.0 Ours 11.1 24.7 30.6 50.7

[0165] Table 3b

[0166] method R@1 R@5 R@10 MedR NoiseEst 6.4 19.8 28.4 39.0 MMT 12.9 29.9 40.1 19.3 Frozen 15.0 30.8 39.8 20.0 Ours 17.8 35.6 44.1 15.5

[0167] Table 3a and Table 3b show the text-to-video retrieval results on the MSR-VTT dataset. The test results of the zero-shot method are shown in Table 3a, and the test results of the training set fine-tuning method are shown in Table 3b. The table includes multiple indicators, including R@1, R@5, R@10, and MedR. Among them, R@K (such as R@1, R@5, and R@10) can be understood as the accuracy of the true result existing in the first K predicted results; MedR can be understood as the Rth true result among multiple predicted results. Higher R@K and lower MedR indicate better results.

[0168] It can be seen from the table that Ours method (i.e., the method proposed in this application) has higher R@K and lower MedR, and achieves better processing results than other methods.

[0169] Table 4a

[0170] method R@1 R@5 R@10 MedR VideoCLIP 16.6 40.9 - - Frozen 21.1 46.0 56.2 7.0 Ours 27.2 50.3 63.6 5.0

[0171] Table 4b

[0172] method R@1 R@5 R@10 MedR HERO 2.1 - 11.4 - CE 16.1 41.1 82.7 8.3 ClipBert 20.4 48.0 60.8 6.0 Frozen 31.0 59.8 72.4 3.0 Ours 36.6 63.9 74.0 3.0

[0173] Table 4a and Table 4b show the text-to-video retrieval results on the DiDeMo dataset. The test results of the zero-shot method are shown in Table 4a, and the test results of the training set fine-tuning method are shown in Table 4b. The table includes multiple indicators, including R@1, R@5, R@10, and MedR. Among them, R@K (such as R@1, R@5, and R@10) can be understood as the accuracy of the true result existing in the first K predicted results; MedR can be understood as the Rth true result among multiple predicted results. Higher R@K and lower MedR indicate better results.

[0174] It can be seen from the table that Ours method (i.e., the method proposed in this application) has higher R@K and lower MedR, and achieves better processing results than other methods.

[0175] Table 5

[0176]

[0177] Table 5 shows the results of zero-shot action recognition on the HMDB51 and UCF101 datasets (the task of zero-shot action recognition can be regarded as video-to-text retrieval). This application uses top1 as the evaluation indicator (top1 can be understood as the accuracy of the first prediction result of the real result). S1, S2, and S3 represent different test partitions, and Mean represents the average results on the three test partitions.

[0178] It can be seen from the table that Ours method (i.e., the method proposed in this application) has a higher Mean and achieves better processing results than other methods.

[0179] The beneficial effects of the present application are as follows: the present application proposes to generate multiple sets of training data using a sample video and the video description text of the sample video, and to obtain multiple sets of sample image sequences corresponding to a sample video by setting different acquisition methods in the process of acquiring the sample image sequence; and to generate multiple mask image sequences corresponding to a set of sample image sequences by masking different image regions in the process of masking the sample image sequence. Through the above method, a large amount of training data can be generated from a small amount of sample videos and their video description texts for pre-training of the model, thereby reducing the amount of data and improving training efficiency.

[0180] The present application also proposes to set an interval threshold for acquiring image frames, or set a random interval range for acquiring image frames, to ensure that the image features of multiple frames in a sampled image sequence have certain differences, thereby improving the effect of image feature extraction.

[0181] The present application also proposes that when acquiring a sampling image sequence and a mask image sequence, mask processing is performed on the same position of each sampling image in the sampling image sequence obtained by interval sampling to obtain a mask image sequence. The above method suppresses the first video encoder from reconstructing the mask area by interpolating the visual content of adjacent frames, so that it needs to perform visual reasoning on the visible video content in the spatial dimension and the temporal dimension, strengthens the data processing model's understanding of the spatial dimension and the temporal dimension, thereby improving the processing effect of the model, and indirectly improving the accuracy of the matching results. In addition, through the interval sampling method, the amount of sampled image data can be reduced while ensuring the processing effect, thereby improving the sampling efficiency of the video to be detected and the efficiency of processing the sampled images.

[0182] The present application also proposes that when inferring the mask area of ​​a target mask image, inference can be performed based on multiple reference mask images, and the multiple reference mask images can be weighted. For example, the weight of the reference mask image in the mask image sequence that is closer to the target mask image is set to a larger weight; the weight of the reference mask image in the mask image sequence that is farther away from the target mask image is set to a smaller weight; or when a distance threshold is met, the reference mask image with a farther distance is ignored, thereby selectively selecting data for mask area inference and improving the accuracy of mask area inference.

[0183] The present application also proposes a variety of methods for obtaining the global features of the mask image sequence. The first method is to fuse the global features corresponding to each mask image in the mask image sequence to obtain the global features of the mask image sequence; the second method is to merge the multiple mask images included in the mask image sequence, and then determine the global features of the mask image sequence based on the merged mask images; the third method is to process the obtained second region feature set to obtain the global features of the mask image sequence. The above methods improve the flexibility of data processing.

[0184] The present application also proposes that the global features are features of the image dimension, and the reference text features are features of the text dimension. By matching the global features and the reference text features, the global features and the reference text features are aligned in the same feature dimension, thereby improving the accuracy and effectiveness of the global loss.

[0185] The present application also proposes that, for multiple local losses corresponding to multiple groups of regional features and matching regional features, the network parameters of the initial first video encoder and the initial text encoder are adjusted iteratively using each local loss. After the network parameters of the first video encoder are adjusted, the network parameters of the second video encoder can be updated according to the adjusted network parameters of the first video encoder, so that the second video encoder aggregates the knowledge of the first video encoder, so that the video features obtained by the second video encoder are close to the text features obtained by the text encoder (that is, the reconstruction target of the mask image obtained by the second video encoder is gradually aligned with the reference text features), thereby improving the accuracy of the model processing; and then the adjusted text encoder and the adjusted first video encoder are jointly trained again in combination with the adjusted second video encoder to obtain a trained text encoder and a trained first video encoder. Through joint training and parameter updating methods, a more accurate matching result of the video description text and the video to be detected can be obtained.

[0186] The present application also improves a specific network parameter updating method, so that the optimization process of the second video encoder is smoother, so that the reconstruction target of the mask image obtained by the second video encoder is gradually aligned with the reference text feature.

[0187] See also Figure 6 , Figure 6 : is a schematic block diagram of a data processing device provided in an embodiment of the present application. The data processing device may specifically include:

[0188] An acquisition module 601 is used to acquire a description text of a video to be detected, and input the description text of the video to be detected into a text encoder for processing to obtain text features of the description text of the video to be detected;

[0189] A processing module 602 is used to input the video image of the video to be detected into a first video encoder for processing to obtain the video features of the video to be detected;

[0190] A matching module 603 is used to determine a matching result between the description text of the video to be detected and the video to be detected according to the text features and the video features;

[0191] Among them, the above-mentioned text encoder and the above-mentioned first video encoder are jointly trained in combination with the second video encoder; during the joint training process, the above-mentioned text encoder is used to obtain reference text features of the video description text of the sample video; the above-mentioned second video encoder is used to obtain the first region feature set of the sampling image sequence of the above-mentioned sample video; the above-mentioned first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the above-mentioned sampling image sequence; the above-mentioned global features and the above-mentioned reference text features are used to determine the global loss, the above-mentioned first region feature set and the above-mentioned second region feature set are used to determine the local loss, the network parameters of the above-mentioned text encoder are adjusted according to the above-mentioned global loss, and the network parameters of the above-mentioned first video encoder are adjusted according to the above-mentioned local loss.

[0192] Optionally, the acquisition module 601 is further used for:

[0193] Obtaining the video description text and the sampled image sequence of the sample video, and obtaining the mask image sequence corresponding to the sampled image sequence;

[0194] The processing module 602 is further used for:

[0195] Input the video description text of the sample video into the initial text encoder for feature extraction to obtain the reference text features;

[0196] Inputting the sampled image sequence into the initial second video encoder for feature extraction to obtain the first region feature set, and inputting the masked image sequence into the initial first video encoder for feature extraction to obtain the global feature and the second region feature set;

[0197] Determine the global loss according to the global feature and the reference text feature, and determine the local loss according to the first regional feature set and the second regional feature set;

[0198] The network parameters of the initial text encoder are adjusted according to the global loss, and the trained text encoder is determined according to the adjusted text encoder; and the network parameters of the initial first video encoder are adjusted according to the local loss, and the trained first video encoder is determined according to the adjusted first video encoder.

[0199] Optionally, the processing module 602, when used to obtain the sampled image sequence of the sample video, is specifically used to:

[0200] Performing interval sampling processing on the sample video to obtain a sampled image sequence; any two sampled images in the sampled image sequence are not adjacent;

[0201] The step of obtaining the mask image sequence corresponding to the sampled image sequence includes:

[0202] Mask processing is performed on each sampled image in the sampled image sequence to obtain the masked image sequence; the masked image regions in each sampled image are matched.

[0203] Optionally, the processing module 602, when used to input the mask image sequence into the initial first video encoder for feature extraction to obtain the second region feature set, is specifically used to:

[0204] Inputting the mask image sequence into an initial first video encoder for processing, performing region division processing on the target mask image in the mask image sequence, and extracting region features of each divided image region;

[0205] wherein the regional features of the first image region including the mask region in the target mask image are predicted features predicted based on the non-masked portion in the reference mask image and the non-masked portion in the target mask image, and the predicted features are close to the features extracted by the initial second video encoder from the second image region matching the first image region in the sampled image sequence; the target mask image is any mask image in the mask image sequence, and the reference mask image is a mask image in the mask image sequence other than the target mask image;

[0206] The second region feature set is obtained according to the region features extracted from each mask image in the mask image sequence.

[0207] Optionally, the first region feature set is composed of region features extracted by the initial second encoder for each sampled image in the sampled image sequence, each region feature corresponds to a divided image region of the sampled image, and the processing module 602, when used to determine the global loss according to the global feature and the reference text feature, and to determine the local loss according to the first region feature set and the second region feature set, is specifically used to:

[0208] Matching the global features with the reference text features, and determining a global loss based on the matched global features and the matched reference text features;

[0209] For any regional feature in the first regional feature set, determining a matching regional feature that matches any regional feature from the second regional feature set;

[0210] Determine a difference parameter between any of the above regional features and the above matching regional features, and determine the above local loss based on the above difference parameter.

[0211] Optionally, the processing module 602, when used to determine the trained text encoder and the first video encoder according to the adjusted text encoder and the first video encoder, is specifically used to:

[0212] After the network parameters of the first video encoder are adjusted, updating the network parameters of the second video encoder according to the adjusted network parameters of the first video encoder to obtain an adjusted second video encoder;

[0213] Combined with the adjusted second video encoder, the adjusted text encoder and the adjusted first video encoder are jointly trained again to obtain a trained text encoder and a trained first video encoder;

[0214] The features obtained by the adjusted second video encoder for the sampled images of the sample video are close to the text features obtained by the adjusted text encoder for the video description text of the sample video.

[0215] Optionally, the video to be detected is included in a video database, the video database includes multiple videos, and the matching module 603 is further used to:

[0216] According to the matching result between each video in the above-mentioned multiple videos and the above-mentioned video description text to be detected, a matching video matching the above-mentioned video description text to be detected is determined from the above-mentioned video database.

[0217] It should be noted that the functions of each functional module of the data processing device in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0218] See also Figure 7 , Figure 7 701, storage device 702 and communication interface 703. The processor 701, storage device 702 and communication interface 703 can exchange data.

[0219] The above-mentioned storage device 702 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the storage device 702 may also include a non-volatile memory (non-volatile memory), such as a flash memory, a solid-state drive (SSD), etc.; the above-mentioned storage device 702 may also include a combination of the above-mentioned types of memory.

[0220] The processor 701 may be a central processing unit (CPU). In one embodiment, the processor 701 may also be a graphics processing unit (GPU). The processor 701 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 702 is used to store program instructions, and the processor 701 may call the program instructions to perform the following operations:

[0221] Obtain a description text of the video to be detected, input the description text of the video to be detected into a text encoder for processing, and obtain text features of the description text of the video to be detected;

[0222] Inputting the video image of the video to be detected into the first video encoder for processing to obtain the video features of the video to be detected;

[0223] Determine a matching result between the description text of the video to be detected and the video to be detected according to the text features and the video features;

[0224] Among them, the above-mentioned text encoder and the above-mentioned first video encoder are jointly trained in combination with the second video encoder; during the joint training process, the above-mentioned text encoder is used to obtain reference text features of the video description text of the sample video; the above-mentioned second video encoder is used to obtain the first region feature set of the sampling image sequence of the above-mentioned sample video; the above-mentioned first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the above-mentioned sampling image sequence; the above-mentioned global features and the above-mentioned reference text features are used to determine the global loss, the above-mentioned first region feature set and the above-mentioned second region feature set are used to determine the local loss, the network parameters of the above-mentioned text encoder are adjusted according to the above-mentioned global loss, and the network parameters of the above-mentioned first video encoder are adjusted according to the above-mentioned local loss.

[0225] Optionally, the processor 701 is further configured to:

[0226] Obtaining the video description text and the sampled image sequence of the sample video, and obtaining the mask image sequence corresponding to the sampled image sequence;

[0227] Input the video description text of the sample video into the initial text encoder for feature extraction to obtain the reference text features;

[0228] Inputting the sampled image sequence into the initial second video encoder for feature extraction to obtain the first region feature set, and inputting the masked image sequence into the initial first video encoder for feature extraction to obtain the global feature and the second region feature set;

[0229] Determine the global loss according to the global feature and the reference text feature, and determine the local loss according to the first regional feature set and the second regional feature set;

[0230] The network parameters of the initial text encoder are adjusted according to the global loss, and the trained text encoder is determined according to the adjusted text encoder; and the network parameters of the initial first video encoder are adjusted according to the local loss, and the trained first video encoder is determined according to the adjusted first video encoder.

[0231] Optionally, the processor 701, when used to obtain the sampled image sequence of the sample video, is specifically used to:

[0232] Performing interval sampling processing on the sample video to obtain a sampled image sequence; any two sampled images in the sampled image sequence are not adjacent;

[0233] The step of obtaining the mask image sequence corresponding to the sampled image sequence includes:

[0234] Mask processing is performed on each sampled image in the sampled image sequence to obtain the masked image sequence; the masked image regions in each sampled image are matched.

[0235] Optionally, the processor 701, when used to input the mask image sequence into an initial first video encoder for feature extraction to obtain the second region feature set, is specifically used to:

[0236] Inputting the mask image sequence into an initial first video encoder for processing, performing region division processing on the target mask image in the mask image sequence, and extracting region features of each divided image region;

[0237] wherein the regional features of the first image region including the mask region in the target mask image are predicted features predicted based on the non-masked portion in the reference mask image and the non-masked portion in the target mask image, and the predicted features are close to the features extracted by the initial second video encoder from the second image region matching the first image region in the sampled image sequence; the target mask image is any mask image in the mask image sequence, and the reference mask image is a mask image in the mask image sequence other than the target mask image;

[0238] The second region feature set is obtained according to the region features extracted from each mask image in the mask image sequence.

[0239] Optionally, the first region feature set is composed of region features extracted by the initial second encoder for each sampled image in the sampled image sequence, each region feature corresponds to a divided image region of the sampled image, and the processor 701, when used to determine the global loss according to the global feature and the reference text feature, and to determine the local loss according to the first region feature set and the second region feature set, is specifically used to:

[0240] Matching the global features with the reference text features, and determining a global loss based on the matched global features and the matched reference text features;

[0241] For any regional feature in the first regional feature set, determining a matching regional feature that matches any regional feature from the second regional feature set;

[0242] Determine a difference parameter between any of the above regional features and the above matching regional features, and determine the above local loss based on the above difference parameter.

[0243] Optionally, the processor 701, when used to determine the trained text encoder and the first video encoder according to the adjusted text encoder and the first video encoder, is specifically used to:

[0244] After the network parameters of the first video encoder are adjusted, updating the network parameters of the second video encoder according to the adjusted network parameters of the first video encoder to obtain an adjusted second video encoder;

[0245] Combined with the adjusted second video encoder, the adjusted text encoder and the adjusted first video encoder are jointly trained again to obtain a trained text encoder and a trained first video encoder;

[0246] The features obtained by the adjusted second video encoder for the sampled images of the sample video are close to the text features obtained by the adjusted text encoder for the video description text of the sample video.

[0247] Optionally, the video to be detected is included in a video database, and the video database includes multiple videos. The processor 701 is further used to:

[0248] According to the matching result between each video in the above-mentioned multiple videos and the above-mentioned video description text to be detected, a matching video matching the above-mentioned video description text to be detected is determined from the above-mentioned video database.

[0249] In a specific implementation, the processor 701, the storage device 702, and the communication interface 703 described in the embodiments of the present application can execute the embodiments of the present application. Figure 2 or Figure 4 The implementation method described in the relevant embodiments of the data processing method provided can also be implemented in the embodiments of the present application Figure 6 The implementation methods described in the relevant embodiments of the provided data processing device will not be repeated here.

[0250] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0251] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device mentioned above, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above Figure 2 , Figure 4 The method in the corresponding embodiment, therefore, will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network, and multiple computer devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0252] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device can perform the above Figure 2 , Figure 4 The method in the corresponding embodiment will therefore not be described in detail here.

[0253] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The above-mentioned storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0254] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of this application are still within the scope of the invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a description text of a video to be detected, inputting the description text of the video to be detected into a text encoder for processing, and obtaining text features of the description text of the video to be detected; Inputting a video image of a video to be detected into a first video encoder for processing to obtain video features of the video to be detected; Determine a matching result between the to-be-detected video description text and the to-be-detected video according to the text features and the video features; Among them, the text encoder and the first video encoder are jointly trained in combination with the second video encoder; during the joint training process, the text encoder is used to obtain reference text features of the video description text of the sample video; the second video encoder is used to obtain the first region feature set of the sample image sequence of the sample video; the first video encoder is used to obtain the global features and the second region feature set of the mask image sequence of the sample image sequence; the global features and the reference text features are used to determine the global loss, the first region feature set and the second region feature set are used to determine the local loss, the network parameters of the text encoder are adjusted according to the global loss, and the network parameters of the first video encoder are adjusted according to the local loss; the first region feature set represents the local features of the sample image sequence, and the second region feature set represents the local features of the mask image sequence.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a video description text and a sampled image sequence of the sample video, and obtaining a mask image sequence corresponding to the sampled image sequence; Inputting the video description text of the sample video into an initial text encoder for feature extraction to obtain the reference text features; Inputting the sampled image sequence into an initial second video encoder for feature extraction to obtain the first region feature set, and inputting the masked image sequence into an initial first video encoder for feature extraction to obtain the global feature and the second region feature set; Determining the global loss according to the global feature and the reference text feature, and determining the local loss according to the first regional feature set and the second regional feature set; The network parameters of the initial text encoder are adjusted according to the global loss, and the trained text encoder is determined according to the adjusted text encoder; and the network parameters of the initial first video encoder are adjusted according to the local loss, and the trained first video encoder is determined according to the adjusted first video encoder.

3. The method according to claim 2, characterized in that Acquiring a sampling image sequence of the sample video, including: Performing interval sampling processing on the sample video to obtain a sampled image sequence; any two sampled images in the sampled image sequence are not adjacent; The step of obtaining a mask image sequence corresponding to the sampled image sequence includes: Mask processing is performed on each sampled image in the sampled image sequence to obtain the masked image sequence; and the masked image regions in each sampled image are matched.

4. The method according to claim 2 or 3, characterized in that: Inputting the mask image sequence into the initial first video encoder to extract features, and obtaining the second region feature set, including: The mask image sequence is input into an initial first video encoder for processing, and for a target mask image in the mask image sequence, the target mask image is subjected to region division processing, and region features of each divided image region are extracted; wherein the region features of a first image region containing a mask region in the target mask image are prediction features obtained based on a non-mask portion in a reference mask image and a non-mask portion in the target mask image, and the prediction features are close to features extracted by the initial second video encoder from a second image region in the sampled image sequence that matches the first image region; the target mask image is any mask image in the mask image sequence, and the reference mask image is a mask image in the mask image sequence other than the target mask image; The second region feature set is obtained according to the region features extracted from each mask image in the mask image sequence.

5. The method according to claim 4, characterized in that The first region feature set is composed of region features extracted by the initial second video encoder for each sampled image in the sampled image sequence, and each region feature corresponds to a divided image region of the sampled image; The determining the global loss according to the global feature and the reference text feature, and determining the local loss according to the first regional feature set and the second regional feature set, comprises: Matching the global feature with the reference text feature, and determining a global loss according to the matched global feature and the matched reference text feature; For any regional feature in the first regional feature set, determining a matching regional feature that matches the any regional feature from the second regional feature set; A difference parameter between any one of the regional features and the matching regional features is determined, and the local loss is determined according to the difference parameter.

6. The method according to claim 2, characterized in that The method further comprises: After the network parameters of the first video encoder are adjusted, updating the network parameters of the second video encoder according to the adjusted network parameters of the first video encoder to obtain an adjusted second video encoder; In combination with the adjusted second video encoder, jointly training the adjusted text encoder and the adjusted first video encoder again to obtain a trained text encoder and a trained first video encoder; The features acquired by the adjusted second video encoder for the sampled images of the sample video are close to the text features acquired by the adjusted text encoder for the video description text of the sample video.

7. The method according to any one of claims 1 to 3 or 6, characterized in that: The video to be detected is contained in a video database, the video database includes a plurality of videos, and the method further includes: According to the matching result between each video in the multiple videos and the description text of the video to be detected, a matching video matching the description text of the video to be detected is determined from the video database.

8. A data processing device, characterized in that: The device comprises: An acquisition module is used to acquire a description text of a video to be detected, input the description text of the video to be detected into a text encoder for processing, and obtain text features of the description text of the video to be detected; A processing module, used for inputting a video image of a video to be detected into a first video encoder for processing to obtain video features of the video to be detected; A matching module is used to determine the matching result between the video description text to be detected and the video to be detected based on the text features and the video features; wherein the text encoder and the first video encoder are obtained by joint training in combination with the second video encoder; during the joint training process, the text encoder is used to obtain the reference text features of the video description text of the sample video; the second video encoder is used to obtain the first regional feature set of the sampled image sequence of the sample video; the first video encoder is used to obtain the global features and the second regional feature set of the mask image sequence of the sampled image sequence; the global features and the reference text features are used to determine the global loss, the first regional feature set and the second regional feature set are used to determine the local loss, the network parameters of the text encoder are adjusted according to the global loss, and the network parameters of the first video encoder are adjusted according to the local loss; the first regional feature set represents the local features of the sampled image sequence, and the second regional feature set represents the local features of the mask image sequence.

9. A computer device, characterized in that: include: A processor, a storage device and a communication interface, wherein the processor, the communication interface and the storage device are connected to each other, wherein the storage device stores an executable program code, and the processor is used to call the executable program code to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions are executed by a processor to implement the data processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, it is used to implement the data processing method according to any one of claims 1 to 7.

Images