Video object positioning method and apparatus, storage medium, program product

Abstract

The application discloses a video object positioning method and device, a storage medium and a program product. After obtaining object information of each target object and a plurality of target texts in a video, a visual feature vector of each target object and a word feature vector in each target text are obtained, a text feature vector of each target text is obtained according to the word feature vector and a graph neural network model, a corresponding fusion feature vector is obtained according to the visual feature vector and the text feature vector, a matching degree between the visual feature vector and the text feature vector is calculated according to the fusion feature vector, and for each target text, a target object corresponding to the target text with the largest matching degree is determined as an object positioning result. The embodiment of the application can not only save the artificial cost of labeling the supervision information of the video frame, but also simultaneously position the video object for a plurality of target texts. The embodiment of the application can be applied to the fields of artificial intelligence, video semantic understanding, intelligent transportation and the like.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a video object positioning method, apparatus, storage medium, and program product. Background Technology

[0002] Video object localization is an important computer vision task used to locate target objects corresponding to target text in videos. In related technologies, fully supervised deep learning methods are commonly used for video object localization. These methods train deep neural network models by annotating each video frame with supervisory information, thereby improving the accuracy of the deep neural network model in locating video objects. However, this method requires a large amount of manual annotation, resulting in significant labor costs. Furthermore, this method can only locate video objects for one target text at a time, and cannot simultaneously locate video objects for multiple target texts, thus hindering the efficiency of video object localization. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This invention provides a video object localization method, apparatus, storage medium, and program product, which can save the manual cost of annotating video frames with supervisory information and can simultaneously locate video objects from multiple target texts, thereby improving the efficiency of video object localization.

[0005] On one hand, embodiments of the present invention provide a video object localization method, including the following steps:

[0006] Obtain object information for each target object in the video;

[0007] The visual feature vectors of each target object are obtained based on the object information of each target object.

[0008] Multiple target texts are obtained, and each word in each target text is vectorized to obtain the word feature vector of each word;

[0009] Based on all the word feature vectors and the pre-trained graph neural network model, obtain the text feature vector of each of the target texts;

[0010] The visual feature vectors and the text feature vectors are fused to obtain multiple fused feature vectors.

[0011] The matching degree between each visual feature vector and each text feature vector is obtained based on each of the fused feature vectors;

[0012] Among all the fused feature vectors corresponding to each target text, the target object corresponding to the one with the highest matching degree is determined as the object localization result.

[0013] On the other hand, embodiments of the present invention also provide a video object positioning device, comprising:

[0014] The object information acquisition unit is used to acquire object information of each target object in the video.

[0015] The first vector acquisition unit is used to acquire the visual feature vector of each target object based on the object information of each target object;

[0016] The second vector acquisition unit is used to acquire multiple target texts, vectorize each word in each target text, and obtain the word feature vector of each word.

[0017] The third vector acquisition unit is used to acquire the text feature vector of each of the target texts based on all the word feature vectors and the pre-trained graph neural network model.

[0018] An information fusion unit is used to fuse the visual feature vectors and the text feature vectors to obtain multiple fused feature vectors.

[0019] A matching calculation unit is used to obtain the matching degree between each visual feature vector and each text feature vector based on each of the fused feature vectors;

[0020] The object localization unit is used to determine the target object with the highest matching degree among all the fused feature vectors corresponding to each target text as the object localization result.

[0021] Optionally, the graph neural network model includes nodes corresponding to the word feature vectors, and the relationships between the nodes correspond to the relationships between the words in the target text; the third vector acquisition unit is further configured to:

[0022] All the word feature vectors are input into the pre-trained graph neural network model for feature extraction, and the global relationship features of all neighboring word feature vectors to each word feature vector are obtained.

[0023] Based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector, the target feature vector of each word is calculated.

[0024] The average value of all the target feature vectors corresponding to each target text is calculated to obtain the text feature vector of each target text.

[0025] Optionally, the third vector acquisition unit is further configured to:

[0026] All the word feature vectors are input into a pre-trained graph neural network model for feature extraction, to obtain the local relation features of each neighboring word feature vector to each word feature vector.

[0027] Based on all the local relation features corresponding to each word feature vector, the global relation features of all the neighboring word feature vectors to each word feature vector are calculated.

[0028] Optionally, the third vector acquisition unit is further configured to:

[0029] Based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector, the intermediate feature vector of each word is calculated.

[0030] The target feature vector of each word is calculated based on the intermediate feature vector of each word and the word feature vector of each word.

[0031] Optionally, the third vector acquisition unit is further configured to:

[0032] The intermediate feature vector of each word and the word feature vector of each word are added together to obtain the word feature sum vector of each word;

[0033] The dimension transformation is performed on the sum of the word features of each word to obtain the target feature vector of each word.

[0034] Optionally, the second vector acquisition unit is further configured to:

[0035] Vector mapping is performed on each word in each of the target texts to obtain the word vectors of each word;

[0036] The word vectors of each of the aforementioned words are feature-encoded to obtain the word feature vectors of each of the aforementioned words.

[0037] Optionally, the object information of the target object corresponds to multiple video frames; the first vector acquisition unit is further configured to:

[0038] The video frames corresponding to the object information of each target object are divided based on the time dimension to obtain multiple video frame segments;

[0039] Obtain the spatiotemporal feature vectors of each target object in each of the video frame segments;

[0040] Based on all the spatiotemporal feature vectors corresponding to each target object, the visual feature vector of each target object is calculated.

[0041] Optionally, the first vector acquisition unit is further configured to:

[0042] In each of the video frames of each video frame segment, the global spatiotemporal features and local spatiotemporal features of each target object are obtained;

[0043] The global spatiotemporal features and the local spatiotemporal features are concatenated to obtain the concatenated feature vector of each target object in each video frame;

[0044] The average value of all the spliced ​​feature vectors of each target object in each of the video frame segments is calculated to obtain the spatiotemporal feature vector of each target object in each of the video frame segments.

[0045] Optionally, the first vector acquisition unit is further configured to:

[0046] Feature encoding is performed on each of the spatiotemporal feature vectors corresponding to each of the target objects to obtain the visual encoding vectors of each of the target objects in each of the video frame segments;

[0047] The average value of all the visual encoding vectors corresponding to each target object is calculated to obtain the visual feature vector of each target object.

[0048] Optionally, the information fusion unit is further configured to:

[0049] The visual feature vectors and the text feature vectors are summed to obtain multiple summed feature vectors;

[0050] The visual feature vectors and the text feature vectors are multiplied by a dot to obtain multiple dot-multiplied feature vectors.

[0051] The corresponding summation feature vectors, dot product feature vectors, visual feature vectors, and text feature vectors are concatenated to obtain multiple fused feature vectors.

[0052] Optionally, the video includes multiple video frames; the object information acquisition unit is further configured to:

[0053] Each object in each of the video frames is detected to obtain multiple object detection boxes;

[0054] Aggregate all the object detection boxes based on the target object to obtain the object information of each target object in the video.

[0055] On the other hand, embodiments of the present invention also provide a video object positioning device, comprising:

[0056] At least one processor;

[0057] At least one memory for storing at least one program;

[0058] The video object localization method as described above is implemented when at least one of the programs is executed by at least one of the processors.

[0059] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the video object positioning method as described above.

[0060] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of a computer device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the computer device to perform the video object positioning method as described above.

[0061] The embodiments of the present invention include at least the following beneficial effects: after acquiring object information of each target object and multiple target texts in the video, visual feature vectors of each target object are obtained based on the object information of each target object, word feature vectors of each word in each target text are vectorized, and text feature vectors of each target text are obtained based on all word feature vectors and a pre-trained graph neural network model. Then, information fusion is performed on each visual feature vector and each text feature vector to obtain multiple fused feature vectors. Next, the matching degree between each visual feature vector and each text feature vector is calculated based on each fused feature vector. Among all fused feature vectors corresponding to each target text, the target object corresponding to the one with the largest matching degree is determined as the object localization result. Since the text feature vectors of the target text are obtained based on word feature vectors and a graph neural network model, and the graph neural network model can associate each target text with its word feature vectors, the text feature vectors of each target text can be enriched by the relationships between them when obtaining the text feature vectors of the target text. This improves the accuracy of the text feature vectors of each target text, and consequently, the accuracy of locating video objects based on the text feature vectors. In addition, after obtaining the visual feature vectors of the target object and the text feature vectors of the target text, the visual feature vectors and text feature vectors are fused to obtain a fused feature vector. Then, the matching degree between the visual feature vectors and text feature vectors is calculated based on the fused feature vector, and the object location result is determined based on the matching degree. Therefore, it is not necessary to use supervision information for video frames, achieving a weak supervision effect, thereby saving the manual cost of annotating video frames with supervision information. Furthermore, since the graph neural network model is used to combine all word feature vectors to obtain the text feature vectors of the target text, it can support the simultaneous location of video objects for multiple target texts, thereby improving the efficiency of video object location.

[0062] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0063] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0064] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of another implementation environment provided by an embodiment of the present invention;

[0066] Figure 3 This is a flowchart of a video object localization method provided in an embodiment of the present invention;

[0067] Figure 4 This is a flowchart illustrating the principle of a video object localization method as a specific example.

[0068] Figure 5 This is a schematic diagram of a video object positioning device provided in an embodiment of the present invention;

[0069] Figure 6 This is a schematic diagram of another video object positioning device provided in an embodiment of the present invention. Detailed Implementation

[0070] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The described embodiments should not be considered as limitations on the present invention, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present invention.

[0071] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0073] Before providing a further detailed description of the embodiments of the present invention, the nouns and terms involved in the embodiments of the present invention will be explained, and the nouns and terms involved in the embodiments of the present invention shall be interpreted as follows.

[0074] 1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0075] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0076] 2) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and further processes the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0077] 3) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0078] 4) Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0079] 5) Intelligent Traffic System (ITS), also known as Intelligent Transportation System, effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing, strengthening the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, improves the environment, and saves energy.

[0080] 6) Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), or vehicle-road cooperative systems for short, represent a development direction for Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information exchange between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active vehicle safety control and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system.

[0081] In related technologies, fully supervised deep learning methods are commonly used for video object localization. These methods train deep neural network models by annotating each video frame with supervisory information, thereby improving the accuracy of object localization. However, this method requires extensive manual annotation, resulting in significant labor costs. To address this issue, some related technologies have proposed weakly supervised deep learning methods for video object localization. These methods transfer multi-instance learning from images to the video domain, treating video object localization as a matching problem between object detection boxes and target objects. Only the text corresponding to the entire video is used as the supervisory signal. The matching degree between each object detection box and the target object is calculated, and candidate boxes matching the target object are selected based on the matching degree. However, both fully supervised and weakly supervised deep learning methods can only locate video objects for one target text at a time, failing to locate video objects for multiple target texts simultaneously, which is detrimental to the efficiency of video object localization.

[0082] To save on the manual cost of annotating video frames with supervisory information and to improve the efficiency of locating video objects by simultaneously locating multiple target texts, this invention provides a video object localization method, a video object localization device, a computer-readable storage medium, and a computer program product. After acquiring object information of each target object and multiple target texts in the video, the method obtains visual feature vectors for each target object based on its object information. It then vectorizes each word in each target text to obtain word feature vectors and obtains text feature vectors for each target text based on all word feature vectors and a pre-trained graph neural network model. Next, it fuses the visual feature vectors and text feature vectors to obtain multiple fused feature vectors. Finally, it calculates the matching degree between each visual feature vector and each text feature vector. Among all the fused feature vectors corresponding to each target text, the target object corresponding to the one with the highest matching degree is determined as the object localization result. Since the text feature vectors of the target text are obtained based on word feature vectors and a graph neural network model, and the graph neural network model can associate each target text with its word feature vectors, the text feature vectors of each target text can be enriched by the relationships between them when obtaining the text feature vectors of the target text. This improves the accuracy of the text feature vectors of each target text, and consequently, the accuracy of locating video objects based on the text feature vectors. In addition, after obtaining the visual feature vectors of the target object and the text feature vectors of the target text, the visual feature vectors and text feature vectors are fused to obtain a fused feature vector. Then, the matching degree between the visual feature vectors and text feature vectors is calculated based on the fused feature vector, and the object location result is determined based on the matching degree. Therefore, it is not necessary to use supervision information for video frames, achieving a weak supervision effect, thereby saving the manual cost of annotating video frames with supervision information. Furthermore, since the graph neural network model is used to combine all word feature vectors to obtain the text feature vectors of the target text, it can support the simultaneous location of video objects for multiple target texts, thereby improving the efficiency of video object location.

[0083] The solutions provided in the embodiments of the present invention involve technologies such as artificial intelligence and video semantic understanding, and are specifically described through the following embodiments.

[0084] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1The implementation environment includes a server 101 and a terminal 102, which are directly or indirectly connected via wired or wireless communication. The server 101 and terminal 102 can be nodes in a blockchain, but this embodiment does not specifically limit their presence.

[0085] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0086] Server 101 may integrate video object positioning function, or server 101 may be equipped with a video object positioning device for implementing video object positioning function.

[0087] Server 101 has at least the following functions: acquiring visual feature vectors of target objects in a video, acquiring text feature vectors of target text, obtaining fused feature vectors based on visual and text feature vectors, and locating objects in the video based on the fused feature vectors. For example, after acquiring object information of each target object in the video and multiple target texts, it can acquire visual feature vectors of each target object based on the object information, acquire text feature vectors of each target text based on each word in each target text, fuse the visual and text feature vectors to obtain multiple fused feature vectors, calculate the matching degree between each visual and text feature vector, and then determine the target object corresponding to the one with the highest matching degree among all fused feature vectors corresponding to each target text as the object location result. Furthermore, after determining the object location result, server 101 can associate the target text corresponding to the object location result with the video, for example, using the target text as the video title or description, and then push the associated target text and video to terminal 102.

[0088] Terminal 102 may include, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. Optionally, users can log in to an internet application platform through terminal 102. The internet application platform can receive relevant target text and videos pushed by server 101, and display or push the target text and videos to the user through terminal 102. The internet application platform may be a social platform, a content interaction platform, an educational platform, or a video sharing platform, etc., without specific limitations here.

[0089] Reference Figure 2 As shown, in one application scenario, server 101 first acquires a video and multiple target texts. Then, for the video, server 101 acquires the object information of each target object in the video and obtains the visual feature vector of each target object based on the object information. For the target texts, server 101 vectorizes each word in each target text to obtain the word feature vector of each word, and obtains the text feature vector of each target text based on all word feature vectors and a pre-trained graph neural network model. Next, server 101 fuses the visual feature vectors and the text feature vectors to obtain multiple fused feature vectors. After obtaining the fused feature vectors, server 101 calculates the matching degree between each visual feature vector and each text feature vector based on each fused feature vector. Then, server 101 determines the target object corresponding to the one with the highest matching degree among all fused feature vectors corresponding to each target text as the object localization result. After obtaining the object localization result, server 101 associates the target text corresponding to the object localization result with the video and uses the target text as the title or description of the video. When the user logs into the social platform through terminal 102, server 101 pushes the associated target text and video to the user through the social platform.

[0090] It should be noted that in various specific embodiments of the present invention, when processing data related to user characteristics, such as user attribute information or sets of attribute information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of the present invention need to obtain user attribute information, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user data for the normal operation of the embodiments of the present invention acquired.

[0091] Figure 3 This is a flowchart of a video object localization method provided in an embodiment of the present invention. In this embodiment, a server is used as the execution subject for illustration. (Refer to...) Figure 3 The video object localization method includes, but is not limited to, steps 110 to 170.

[0092] Step 110: Obtain object information for each target object in the video.

[0093] In this step, the server can first retrieve the pre-stored video from the database, and then obtain the object information of each target object in the video. This allows subsequent steps to obtain the visual feature vectors of each target object based on this object information. The video consists of multiple video frames, typically expressed in frames per second (FPS). Each video frame is a still image; when multiple video frames are played sequentially, moving images, i.e., video content, can be created. The video can include multiple objects, and correspondingly, each video frame will include multiple objects. Objects in the video can be people, animals, plants, scenery, or items, etc., without specific limitations. The target object can be at least one of the multiple objects in the video.

[0094] In some possible implementations, the object information of the target object may include the position information of the target object in the video frame, such as the coordinate information of the target object in the video frame, the spatial range occupied by the target object in the video frame, etc., which are not specifically limited here.

[0095] In some possible implementations, when acquiring object information of each target object in a video, multiple object detection boxes can be obtained by first detecting each object in each video frame. Then, all object detection boxes are aggregated based on the target object to obtain the object information of each target object in the video. Specifically, when obtaining the object detection boxes, the coordinates of the center point of the object detection box in the video frame, as well as the length and height of the object detection box in the video frame, can be obtained. In other words, the object information of each object in each video frame can be obtained. Then, by aggregating all object detection boxes based on the target object, the object information of each target object in the video can be obtained. The object information of each target object in the video can be represented by a sequence of detection boxes composed of multiple object detection boxes. Each object detection box in this sequence can be represented by a 4-dimensional vector. This 4-dimensional vector can include the horizontal coordinates of the center point of the object detection box in the video frame, the vertical coordinates of the center point of the object detection box in the video frame, the length of the object detection box in the video frame, and the height of the object detection box in the video frame. Therefore, a target object can be considered as an object determined by connecting object detection boxes corresponding to the same object in consecutive video frames. When detecting objects in various video frames to obtain multiple object detection boxes, an object detection model can be used to detect each object in each video frame, resulting in object detection boxes for each object in each video frame. After obtaining multiple object detection boxes for each object, one or more target objects can be identified among these objects. Then, all object detection boxes are aggregated based on the target objects to obtain object information for each target object. In the process of aggregating all object detection boxes based on target objects to obtain object information for each target object, the aggregation can be based on the appearance features of the target objects and the positional differences of the target objects in different video frames. Here, the appearance features of the target objects refer to the external features of the target objects, such as shape and size; the positional differences of the target objects in different video frames are used to reflect the motion information of the target objects. That is to say, when aggregating all object detection boxes, between adjacent video frames, the object detection boxes corresponding to target objects with consistent appearance features and close positions are aggregated, thereby obtaining more accurate object information for each target object.

[0096] It should be noted that the object detection model can be Faster R-CNN or YOLO (You Only LookOnce), etc., and the appropriate choice can be made according to the actual application. No specific limitation is made here.

[0097] The Faster-RCNN model uses Convolutional Neural Networks (CNNs) to generate reference boxes and shares this CNN with the object detection network, which greatly reduces the number of reference boxes, thereby improving detection efficiency and accuracy.

[0098] The YOLO model applies a single CNN to the entire image, dividing it into multiple grids and predicting the class probability and the size of the bounding box for each grid. The YOLO model predicts the probability of an object being detected within the bounding box. If this probability is greater than or equal to a preset probability threshold, the corresponding object is considered identified; otherwise, it is considered that no object has been identified.

[0099] The following example illustrates the process of obtaining object information for each target object in a video.

[0100] For a video V = {v t} T t＝1 , where v t It refers to the video frames in video V, where T is the number of video frames. In other words, video V includes video frame v1, video frame v2, ..., video frame v... T-1 and video frame v T When acquiring object information for each target object in video V, the Faster-RCNN model is used to detect M target objects (M is a positive integer) in each video frame, obtaining object detection boxes for the M target objects in each video frame. Each object detection box can be represented by a 4-dimensional vector, which includes the horizontal coordinate of the object detection box's center point in the video frame, the vertical coordinate of the object detection box's center point in the video frame, the length of the object detection box in the video frame, and the height of the object detection box in the video frame. After obtaining the object detection boxes for the M target objects in each video frame, the SORT algorithm is used to aggregate all object detection boxes based on the target object, obtaining object information for the M target objects. This object information for the M target objects can be represented as {p m} M m＝1 , where p m This represents the object information of the m-th target object in the video. The object information is a sequence of detection boxes composed of multiple object detection boxes represented by a 4-dimensional vector. The SORT algorithm is an algorithm for associating targets, enabling the correlation detection of target objects between video frames.

[0101] Step 120: Obtain the visual feature vector of each target object based on the object information of each target object.

[0102] In this step, since the object information of each target object in the video was obtained in step 110, the visual feature vector of each target object can be obtained based on the object information of each target object, so that subsequent steps can calculate the degree of matching between the target object and the target text based on the visual feature vector of each target object.

[0103] In some possible implementations, when the object information of a target object corresponds to multiple video frames, when performing the step of obtaining the visual feature vector of each target object based on its object information, the multiple video frames corresponding to the object information of each target object can first be divided based on the time dimension to obtain multiple video frame segments. Then, the spatiotemporal feature vector of each target object in each video frame segment can be obtained. Next, the visual feature vector of each target object can be calculated based on all the spatiotemporal feature vectors corresponding to each target object. Here, the object information of the target object is obtained based on object detection boxes in consecutive video frames. Therefore, the multiple consecutive video frames corresponding to the object information of the target object can be divided based on the same duration to obtain multiple video frame segments with the same duration. Since the multiple video frame segments corresponding to the object information of the same target object have the same duration, the spatiotemporal feature vector of the target object in each video frame segment can be obtained based on the same processing parameters, thereby improving the efficiency of obtaining the spatiotemporal feature vector of the target object in each video frame segment. It is worth noting that since the duration of different target objects appearing in the video is not necessarily the same, the number of video frames corresponding to different target objects is not necessarily the same; that is, the video frame segments corresponding to different target objects will have different durations. After obtaining multiple video frame segments, the spatiotemporal feature vectors of each target object in each video frame segment can be acquired. Then, the visual feature vector of each target object can be calculated based on all the spatiotemporal feature vectors corresponding to each target object. The spatiotemporal feature vector of a target object reflects both its spatial and temporal dimensions within the video frame. Therefore, calculating the visual feature vector of a target object based on its spatiotemporal feature vectors can improve the accuracy of the visual feature vectors, thereby enhancing the accuracy of subsequent steps in locating video objects based on their visual feature vectors.

[0104] In some possible implementations, when performing the step of obtaining the spatiotemporal feature vectors of each target object in each video frame segment, the global and local spatiotemporal features of each target object can be obtained first in each video frame of each video frame segment. Then, the global and local spatiotemporal features are concatenated to obtain the concatenated feature vectors of each target object in each video frame. Next, the average value of all concatenated feature vectors of each target object in each video frame segment is calculated to obtain the spatiotemporal feature vectors of each target object in each video frame segment. Specifically, when obtaining the global and local spatiotemporal features of each target object, the object information of each target object can be input into a feature extraction model for feature extraction, thereby obtaining the global and local spatiotemporal features of each target object. It should be noted that global spatiotemporal features refer to the spatiotemporal feature information of the target object within the entire video frame, reflecting the spatial and temporal dimensions of the target object within the entire video frame; local spatiotemporal features refer to the spatiotemporal feature information of the target object within a certain local area of ​​the video frame, reflecting the spatial and temporal dimensions of the target object within that local area of ​​the video frame. For example, in one embodiment, the object information of the target object can be input into an Inflated 3D ConvNet (I3D ConvNet) model for feature extraction. In this case, the RGB (Red, Green, Blue) information output by the I3D ConvNet model for the target object is the global spatiotemporal feature of the target object. Then, Region of Interest (ROI) alignment is performed on the global spatiotemporal feature of the target object to obtain the local spatiotemporal feature of the target object. Alternatively, in other embodiments, other neural network models (such as different deep convolutional neural network models) can be used to obtain the global and local spatiotemporal features of the target object; no specific limitations are made here. The Inflated 3D ConvNet model achieves seamless learning of spatiotemporal features by extending the convolutional and pooling kernels in a very deep image classification network from 2D to 3D. Region of Interest alignment is a method of dividing the feature map and then using bilinear interpolation to align the regions. ROI alignment ensures a one-to-one correspondence between pixels in the preceding and following inputs during the pooling process. After obtaining the global and local spatiotemporal features of each target object, the global and local spatiotemporal features can be concatenated end to end to obtain the concatenated feature vector of each target object in each video frame. Then, the average value of all concatenated feature vectors of each target object in each video frame segment is calculated to obtain the spatiotemporal feature vector of each target object in each video frame segment.

[0105] The following example illustrates the process of obtaining the spatiotemporal feature vector of a target object.

[0106] Suppose we obtain object information {p} of M target objects. m} M m＝1 Then, for each target object, the multiple video frames corresponding to the object information are divided equally according to the time length to obtain... There are N video frame segments, where N is a positive integer. For each video frame in each video frame segment, a dilated 3D convolutional network model is first used to extract global spatiotemporal features of each target object in each video frame. Then, the global spatiotemporal features of each target object are aligned with regions of interest to obtain local spatiotemporal features of each target object. Finally, the global spatiotemporal features and local spatiotemporal features are concatenated to obtain a length of [length missing]. The concatenated feature vectors are then calculated, and the average of all concatenated feature vectors for each target object in each video frame segment is calculated to obtain the spatiotemporal feature vector for each target object in each video frame segment. The spatiotemporal feature vector can be represented as...

[0107] In some possible implementations, when performing the step of calculating the visual feature vector of each target object based on all spatiotemporal feature vectors corresponding to each target object, feature encoding can be performed on each spatiotemporal feature vector corresponding to each target object first to obtain the visual encoded vector of each target object in each video frame segment. Then, the average value of all visual encoded vectors corresponding to each target object is calculated to obtain the visual feature vector of each target object. Specifically, when performing feature encoding on the spatiotemporal feature vectors, the encoder in the Transformer model can be used. The Transformer model is a neural network model for natural language processing. The Transformer model mainly includes an encoder and a decoder. The encoder maps natural language sequences to hidden layer information, while the decoder maps the hidden layer information back to natural language sequences. The encoder of the Transformer model includes interconnected multi-head self-attention mechanism modules and feedforward network modules. Self-attention mechanism refers to using other features to represent each feature to obtain an enhanced semantic vector for that feature, where the weights of the other features in representing this feature are different. Multi-head self-attention mechanism refers to using multiple self-attention mechanisms to capture relationships at different abstract levels from different perspectives. When using a Transformer model encoder to encode spatiotemporal feature vectors, the encoding process can be represented by the following formula (1):

[0108]

[0109] Where Q1, K1, and V1 are all input parameters of formula (1), and the values ​​of Q1, K1, and V1 are the same; E p () represents the encoder function; and All are transformation matrices that need to be trained; O p This is the output of the encoder (e.g., a visual encoding vector). After training the encoder, the trained output can be obtained. and At this point, by combining formula (1), feature encoding processing of spatiotemporal feature vectors can be achieved.

[0110] The following example illustrates the process of obtaining the visual feature vector of a target object.

[0111] Assuming that the spatiotemporal feature vectors of each target object in each video frame segment are obtained... Subsequently, an object-level encoder can be used to process the spatiotemporal feature vectors. Perform feature encoding, for example, converting spatiotemporal feature vectors Inputting Q1, K1, and V1 into formula (1) respectively, we can obtain the following formula (2):

[0112]

[0113] in, Let be the visual encoding vector of the m-th target object. By solving formula (2), the visual encoding vector of the target object can be obtained. Then, by calculating the average value of all visual encoding vectors corresponding to each target object, the visual feature vector of each target object can be obtained.

[0114] Step 130: Obtain multiple target texts, vectorize each word in each target text, and obtain the word feature vector of each word.

[0115] In this step, the server can first retrieve multiple pre-stored target texts from the database, and then vectorize each word in each target text to obtain the word feature vector of each word. This allows subsequent steps to obtain the text feature vector of each target text based on the corresponding word feature vector. These target texts can be texts corresponding to the video in step 110, or texts not corresponding to the video in step 110; no specific limitation is made here.

[0116] In some possible implementations, when vectorizing individual words in each target text, pre-trained models such as word2vec, semantic representation models (Embeddings from Language Models, ELMo), OpenAI-GPT, or Bidirectional Encoder Representation from Transformer (BERT) can be used to convert each word in the target text into a vector representation. Word2vec, ELMo, OpenAI-GPT, and BERT are all common models in the field of natural language processing. For the principles and structures of Word2vec, ELMo, OpenAI-GPT, and BERT models, please refer to the relevant descriptions in related technologies; they will not be elaborated upon here.

[0117] In some possible implementations, in the process of vectorizing each word in each target text to obtain the word feature vector of each word, the word vector of each word in each target text can be obtained by first performing vector mapping on each word in each target text, and then performing feature encoding on the word vector of each word to obtain the word feature vector of each word. Among them, in the process of performing vector mapping on words to obtain word vectors, pre-trained models such as Word2vec or BERT model can be used to perform vector mapping on words. After obtaining the word vectors of each word, the encoder in the Transformer model can be used to perform feature encoding on the word vectors of each word, thereby obtaining the word feature vector of each word. When using the encoder of the Transformer model to perform feature encoding on the word vectors of words, the encoding process can be represented by the following formula (3):

[0118]

[0119] Where Q2, K2, and V2 are all input parameters of formula (3), and the values ​​of Q2, K2, and V2 are the same; E s () represents the encoder function; and Both are transformation matrices that need to be trained; h is the encoder's output (e.g., word feature vectors). After training the encoder, the trained output can be obtained. and At this point, by combining formula (3), the feature encoding processing of word vectors can be achieved.

[0120] The following example illustrates the process of obtaining word feature vectors.

[0121] Assuming L target texts are obtained Among them, s i Let i be the i-th target text. In this case, a pre-trained BERT-Base model can be used to map the vectors of each word in the target text, obtaining the word vectors for each word. The word vectors can be represented as... Its dimensions are It's important to note that the BERT-Base model is a type of BERT model. Compared to other types of BERT models, the BERT-Base model has different model parameters. For example, the BERT-Base model may include 12 Transformer modules, 768 hidden layer units, and 12 attention heads. After obtaining the word vectors for each word, a word-level encoder can be used to process the word vectors. Perform feature encoding, for example, by encoding word vectors. Inputting Q2, K2, and V2 into formula (3) respectively, we can obtain the following formula (4):

[0122]

[0123] By solving formula (4), the word feature vector can be obtained.

[0124] Step 140: Obtain the text feature vector of each target text based on the feature vectors of all words and the pre-trained graph neural network model.

[0125] Multiple target texts typically contain common words. If these common words can be used as linking points to further mine the mutual information between these target texts, more accurate text feature vectors can be obtained for each target text. To achieve this, this step uses all word feature vectors and a pre-trained graph neural network model to link the target texts, thereby obtaining more accurate text feature vectors for each target text. This allows subsequent steps to calculate the matching degree between the target text and the target object based on the text feature vectors of each target text. The graph neural network model uses neural networks to learn graph-structured data, extracting and mining features and patterns within the graph-structured data to satisfy graph learning tasks such as clustering, classification, prediction, segmentation, and generation. Graph neural network models can include Graph Convolutional Networks (GCNs), Graph Auto-encoders (GAEs), Graph Generative Networks (GGNs), and Graph Attention Networks (GATs). GCNs determine the representation of neighboring nodes based on the spatial features of graph data, making the representations of each node's neighbors uniform and regular, thus facilitating convolution operations. Graph Auto-encoders can learn information about graph nodes in a semi-supervised or unsupervised manner. Graph Generative Networks can recombine nodes and edges using specific rules to generate target graphs with specific attributes and requirements. Graph Attention Networks focus on nodes and edges more relevant to the task, thereby improving training effectiveness and testing accuracy.

[0126] It should be noted that the graph neural network model includes nodes corresponding to word feature vectors, and the relationships between nodes correspond to the relationships between words in the target text. Therefore, in some possible implementations, when obtaining the text feature vectors of each target text based on all word feature vectors and the pre-trained graph neural network model, all word feature vectors can first be input into the pre-trained graph neural network model for feature extraction. This obtains the global relationship features of all neighboring word feature vectors to each word feature vector. Then, based on the global relationship features corresponding to each word feature vector and all neighboring word feature vectors, the target feature vector of each word is calculated. Finally, the average value of all target feature vectors corresponding to each target text is calculated to obtain the text feature vector of each target text. It should also be noted that for a given word feature vector, the global relationship features of all neighboring word feature vectors represent the degree of influence of all neighboring word feature vectors on the given word feature vector. The larger the value of this global relationship feature, the higher the importance of all neighboring word feature vectors to the given word feature vector. In the process of inputting all word feature vectors into a pre-trained graph neural network model for feature extraction to obtain the global relationship features of all neighboring word feature vectors to each word feature vector, we can first input all word feature vectors into the pre-trained graph neural network model for feature extraction to obtain the local relationship features of each neighboring word feature vector to each word feature vector. Then, based on all the local relationship features corresponding to each word feature vector, we can calculate the global relationship features of all neighboring word feature vectors to each word feature vector. It should be noted that, for a certain word feature vector, the local relationship feature of a certain neighboring word feature vector to that word feature vector represents the degree of influence of that neighboring word feature vector on that word feature vector. The larger the value of the local relationship feature, the higher the importance of that neighboring word feature vector to that word feature vector. In addition, when calculating the target feature vector of each word based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector, the intermediate feature vector of each word can be calculated first based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector. Then, the target feature vector of each word can be calculated based on the intermediate feature vector of each word and the word feature vector of each word.In calculating the target feature vector of each word based on the intermediate feature vector and the word feature vector of each word, the intermediate feature vector and the word feature vector of each word can be added together to obtain the word feature sum vector of each word. Then, the dimension of the word feature sum vector of each word is transformed to obtain the target feature vector of each word.

[0127] The following example illustrates the process of obtaining the text feature vector of a target text.

[0128] Suppose a graph attention network model has been pre-trained, and the nodes of this graph attention network model are L target text nodes. The graph attention network model connects multiple graph convolutional layers through a dense connection pattern, creating dense connections between the words. This dense connection pattern means that for each graph convolutional layer in the graph attention network model, the input features of the current convolutional layer are the concatenated aggregated features of the input and output features of the previous convolutional layer. Once the word feature vector h of each word is obtained... i Next, the word feature vector h of each word is... i The input is fed into the corresponding node in the graph attention network model (for ease of description, the node is also referred to as h). i For each current node, calculate the local relation features of each neighboring node to the current node (i.e., the local relation features of each neighboring word feature vector to the current word feature vector), where the local relation features of neighboring nodes to the current node can be represented by the following formula (5):

[0129] z ij =f(W a [W q h i |W k h j (5)

[0130] Among them, z ij It is the adjacent node h j For the current node h i Local relational features, W a W q and W k Both are transformation matrices after training, | indicates concatenation operation, and f() represents the activation function. For example, f) can be the LeakyReLU activation function.

[0131] After calculating the local relation features of each neighboring node to the current node, the global relation features α of all neighboring nodes to the current node are calculated using the following formula (6). ij :

[0132]

[0133] After calculating the global relationship feature α between all neighboring nodes and the current node. ij Then, the intermediate feature vector u of each word is calculated using the following formula (7). i :

[0134]

[0135] Among them, W v The transformation matrix can be obtained through training; σ() represents the nonlinear transformation function.

[0136] The intermediate feature vector u of each word is calculated. i Then, to avoid the gradient vanishing problem, we can adjust the intermediate feature vector u. i Residual learning is performed to obtain the word feature sum vector. Specifically, the intermediate feature vector u of the word can be obtained by the following formula (8). i and the word feature vector h i Add them together to obtain the word feature sum vector h. i ′:

[0137] h i ′=u i +h i (8)

[0138] The word feature sum vector h of each word is calculated. i After that, the word feature vector h of each word can be summed. i Perform a dimensionality transformation to obtain the target feature vector for each word. For example, suppose the word feature summation vector is h. i The dimension of ' is 128, and a linear transformation is used as a feedforward neural network layer to sum the word feature vector h. i A dimensional transformation is performed to obtain a 64-dimensional target feature vector. After obtaining the target feature vectors of each word, the average value of all target feature vectors corresponding to each target text is calculated to obtain the text feature vector of each target text. The text feature vector of the target text can be calculated using the following formula (9):

[0139]

[0140] in, Let be the text feature vector of the i-th target text; h′ represents the number of words contained in the i-th target text. i，rLet be the target feature vector of the r-th word in the i-th target text.

[0141] Step 150: Fuse the visual feature vectors and text feature vectors to obtain multiple fused feature vectors.

[0142] In this step, since the visual feature vectors of each target object were obtained in step 120 and the text feature vectors of each target text were obtained in step 140, the visual feature vectors and text feature vectors can be fused to obtain multiple fused feature vectors. This allows subsequent steps to calculate the degree of matching between the target text and the target object based on the fused feature vectors.

[0143] In some possible implementations, when fusing visual feature vectors and text feature vectors to obtain multiple fused feature vectors, the visual and text feature vectors can be summed first to obtain multiple summed feature vectors. Then, the visual and text feature vectors can be multiplied to obtain multiple multiplied feature vectors. Finally, the corresponding summed feature vectors, multiplied feature vectors, visual feature vectors, and text feature vectors are concatenated to obtain multiple fused feature vectors. Since the fused feature vectors are obtained after summing and multiplying visual and text feature vectors, they can more accurately express the relationship between the target text and the target object. This improves the accuracy of the matching degree between the visual and text feature vectors calculated based on the fused feature vectors in subsequent steps, thereby improving the accuracy of the final object localization result.

[0144] The following example illustrates the process of obtaining the fused feature vector.

[0145] Assume the visual feature vector of the m-th target object is The obtained text feature vector of the i-th target text is Therefore, when obtaining the fused feature vector, the summation feature vector and the dot product feature vector can be calculated using the following formulas (10) and (11):

[0146]

[0147]

[0148] in, The summed feature vector is obtained by summing the text feature vector of the i-th target text and the visual feature vector of the m-th target object. The dot product feature vector is obtained by multiplying the text feature vector of the i-th target text with the visual feature vector of the m-th target object.

[0149] After obtaining the summed eigenvector and the dot product eigenvector, the fused eigenvector is obtained using the following formula (12):

[0150]

[0151] in, Let be the fused feature vector of the i-th target text and the m-th target object; || denotes the concatenation operation.

[0152] Step 160: Obtain the matching degree between each visual feature vector and each text feature vector based on each fused feature vector.

[0153] In this step, since the fused feature vector between each visual feature vector and each text feature vector was obtained in step 150, the matching degree between each visual feature vector and each text feature vector can be obtained based on each fused feature vector, so that subsequent steps can determine the object localization result based on the matching degree between each visual feature vector and each text feature vector.

[0154] In related technologies, such as commonly used fully supervised video object localization methods, Euclidean distance or cosine distance is often used to calculate the matching degree between visual feature vectors and text feature vectors. However, Euclidean distance or cosine distance requires two input parameters. When the input parameter is only the fused feature vector, commonly used Euclidean distance or cosine distance methods cannot support calculating the matching degree between visual feature vectors and text feature vectors. To solve this problem, this step introduces a fully connected network model, which includes two or more fully connected layers. After training this fully connected network model, the fused feature vector is input into the fully connected network model to obtain the matching degree between the visual feature vector and the text feature vector.

[0155] In some possible implementations, taking a fully connected network model consisting of two fully connected layers as an example, the process by which the fully connected network model obtains the matching degree between the visual feature vector and the text feature vector based on the fused feature vector can be represented by the following formula (13):

[0156] D = tanh(W2tanh(W1v)) (13)

[0157] Where D is the matching degree between the visual feature vector and the text feature vector, W1 is the weight of the first fully connected layer in the fully connected network model, W2 is the weight of the second fully connected layer in the fully connected network model, tanh() is the activation function, and v is the input parameter of the fully connected network model.

[0158] When using a fully connected network model to obtain the matching degree between visual feature vectors and text feature vectors, the fully connected network model can be initialized randomly according to a normal distribution. Then, the fully connected network model can be trained, and the weights W1 of the first fully connected layer and W2 of the second fully connected layer can be continuously updated until the network loss converges. At this point, the fused feature vector is input into the input parameter v of the fully connected network model for calculation, and the matching degree D between the visual feature vector and the text feature vector can be calculated.

[0159] Step 170: Among all the fused feature vectors corresponding to each target text, the target object corresponding to the one with the highest matching degree is determined as the object localization result.

[0160] Since different target texts may be associated with the same target object, that is, for the same target object, multiple target texts may be calculated to match it. In order to obtain more accurate object localization results, in this step, the target object corresponding to the one with the highest matching degree among all the fusion feature vectors corresponding to each target text can be determined as the object localization result. That is, for each target text, the highest matching degree is determined among all the matching degrees corresponding to that target text, and the corresponding target fusion feature vector is determined based on the highest matching degree. Then, the corresponding target object is determined based on the target fusion feature vector, and then the target object is determined as the object localization result corresponding to that target text.

[0161] In some possible implementations, when there are multiple maximum matching degrees, a target matching degree can be randomly selected from these maximum matching degrees, and the corresponding target fusion feature vector can be determined based on the target matching degree. Then, the corresponding target object can be determined based on the target fusion feature vector, and then the target object can be determined as the object location result corresponding to the target text.

[0162] In this embodiment, the video object localization method, including steps 110 to 170 above, obtains object information of each target object and multiple target texts in the video. It then obtains visual feature vectors of each target object based on its object information, vectorizes each word in each target text to obtain word feature vectors, and obtains text feature vectors of each target text based on all word feature vectors and a pre-trained graph neural network model. Next, it fuses the visual feature vectors and text feature vectors to obtain multiple fused feature vectors. Finally, it calculates the matching degree between each visual feature vector and each text feature vector based on the fused feature vectors. Among all the fused feature vectors corresponding to each target text, the target object corresponding to the one with the highest matching degree is determined as the object localization result. Since the text feature vectors of the target text are obtained based on word feature vectors and a graph neural network model, and the graph neural network model can associate each target text with its word feature vectors, the text feature vectors of each target text can be enriched by the relationships between them when obtaining the text feature vectors of the target text. This improves the accuracy of the text feature vectors of each target text, and consequently, the accuracy of locating video objects based on the text feature vectors. In addition, after obtaining the visual feature vectors of the target object and the text feature vectors of the target text, the visual feature vectors and text feature vectors are fused to obtain a fused feature vector. Then, the matching degree between the visual feature vectors and text feature vectors is calculated based on the fused feature vector, and the object location result is determined based on the matching degree. Therefore, it is not necessary to use supervision information for video frames, achieving a weak supervision effect, thereby saving the manual cost of annotating video frames with supervision information. Furthermore, since the graph neural network model is used to combine all word feature vectors to obtain the text feature vectors of the target text, it can support the simultaneous location of video objects for multiple target texts, thereby improving the efficiency of video object location.

[0163] In some possible implementations, before using a fully connected network model to obtain the matching degree between visual feature vectors and text feature vectors, the matching degree between text and video can be used as supervision information to train the fully connected network model. This supervision information does not involve information on the time axis, but only whether the text and video match. Since this supervision information does not involve the matching degree between the text and the objects in the video, it can achieve the effect of weak supervision, thereby saving the manual cost of annotating video frames with supervision information.

[0164] In some possible implementations, when training a fully connected network model, ranking loss and mutual exclusion loss can be used as training objectives until the trained fully connected network model simultaneously meets the requirements of both ranking loss and mutual exclusion loss. The ranking loss requirement refers to the matching degree of positive sample pairs being greater than that of negative sample pairs during the training process. A positive sample pair is one where the matching degree obtained by the fully connected network model corresponds to the actual matching degree, while a negative sample pair is one where the matching degree obtained by the fully connected network model does not correspond to the actual matching degree. For example, after inputting the fused feature vectors corresponding to matched text and video training samples into the fully connected network model, if the matching degree output by the fully connected network model is greater than or equal to a preset matching threshold, it indicates that the text and video training samples are positive sample pairs; conversely, after inputting the fused feature vectors corresponding to mismatched text and video training samples into the fully connected network model, if the matching degree output by the fully connected network model is less than a preset matching threshold, it indicates that the text and video training samples are negative sample pairs. The requirement of mutual exclusion loss means that during the training of a fully connected network model, for each text training sample, the matching degree between it and multiple target objects in the video training samples conforms to a corresponding distribution, and the distributions corresponding to each text training sample have significant differences. These differences can be described by Kullback-Leibler Divergence (KL Divergence). KL Divergence measures the degree of difference between two distributions; the smaller the difference, the smaller the KL Divergence; the larger the difference, the larger the KL Divergence; and if the two distributions are identical, the KL Divergence is 0. It should be noted that this embodiment can support simultaneous localization of video objects for multiple target texts. However, generally, each target text can only be matched with one target object. Therefore, when training the fully connected network model, the mutual exclusion relationship between the matching results of target texts and different target objects needs to be used as the training objective.

[0165] It should be noted that ranking loss measures the matching relationship between text training samples and video training samples, while mutual exclusion loss measures the relationship between target objects in text training samples and video training samples. Therefore, using a fully connected network model that meets the requirements of both ranking loss and mutual exclusion loss can more accurately obtain the matching degree between target text and video, thereby improving the accuracy of the final object localization result.

[0166] The video object positioning method of this invention will be described in detail below with specific examples.

[0167] Reference Figure 4As shown, Figure 4 This is a flowchart illustrating the principle of a video object localization method as a specific example. Figure 4 In this paper, the video object localization method can be divided into four main processing parts: feature extraction for video, feature extraction for target text, cross-modal information fusion, and localization operation.

[0168] Feature extraction from videos can include the following steps: For a given video, firstly, perform detection, aggregation, and concatenation on each object in each video frame to obtain object information for each target object in the video; then, divide the multiple video frames corresponding to the object information of each target object into multiple video frame segments based on the time dimension; for each video frame segment, obtain the global and local spatiotemporal features of each target object in each video frame, and concatenate the global and local spatiotemporal features to obtain the spatiotemporal feature vector of each target object in each video frame segment; then, use the encoder in the object-level Transformer model to perform feature encoding based on the self-attention mechanism on the spatiotemporal feature vector of each target object to obtain the visual encoding vector, and calculate the average value of all visual encoding vectors corresponding to each target object to obtain the visual feature vector of each target object.

[0169] Feature extraction for target text can include the following steps: For a given set of target texts, firstly, use a pre-trained BERT model to extract word vectors for each word in each target text; then, use the encoder in a word-level Transformer model to perform feature encoding on the word vectors of each word based on a self-attention mechanism to obtain word feature vectors for each word; then, use a graph attention network model to fully mine the information between words and target texts to obtain target feature vectors for each word; finally, calculate the average of all target feature vectors corresponding to each target text to obtain the text feature vectors for each target text.

[0170] Cross-modal information fusion may include the following steps: summing each visual feature vector and each text feature vector to obtain multiple summed feature vectors; multiplying each visual feature vector and each text feature vector to obtain multiple multiplied feature vectors; and concatenating the corresponding summed feature vectors, multiplied feature vectors, visual feature vectors, and text feature vectors to obtain multiple fused feature vectors.

[0171] The localization operation may include the following steps: first, obtain the matching degree between each visual feature vector and each text feature vector based on each fused feature vector; then, among all the fused feature vectors corresponding to each target text, determine the target object corresponding to the one with the highest matching degree as the object localization result.

[0172] The following examples illustrate the application scenarios of the embodiments of the present invention.

[0173] It should be noted that the video object positioning method provided in this embodiment of the invention can be applied to different application scenarios, such as matching text to videos and searching for videos based on text. The following description uses the scenarios of matching text to videos and searching for videos based on text as examples.

[0174] Scene 1

[0175] The video object localization method provided in this invention can be applied to scenarios of matching text to videos. Specifically, after the server obtains the video and multiple target texts from the database, for the video, the server first detects each object in each video frame to obtain multiple object detection boxes, and then aggregates all object detection boxes based on target objects to obtain object information of each target object in the video. Next, the server divides the multiple video frames corresponding to the object information of each target object based on the time dimension to obtain multiple video frame segments. Then, in each video frame of each video frame segment, the server obtains the global and local spatiotemporal features of each target object. The global and local spatiotemporal features are then concatenated to obtain the concatenated feature vector of each target object in each video frame. After obtaining the concatenated feature vector of each target object in each video frame, the server calculates the average value of all concatenated feature vectors of each target object in each video frame segment to obtain the spatiotemporal feature vector of each target object in each video frame segment. Then, the server performs feature encoding on each spatiotemporal feature vector corresponding to each target object to obtain the visual encoding vector of each target object in each video frame segment. Finally, the server calculates all visual encoding vectors corresponding to each target object. The average value of the code vectors is used to obtain the visual feature vectors of each target object. For multiple target texts, the server first performs vector mapping on each word in each target text to obtain the word vectors of each word, and performs feature encoding on each word vector to obtain the word feature vectors of each word. Then, all word feature vectors are input into a pre-trained graph neural network model for feature extraction to obtain the local relation features of each word feature vector to each word feature vector. Based on all the local relation features corresponding to each word feature vector, the global relation features of all the neighboring word feature vectors to each word feature vector are calculated. Then, based on the global relation features corresponding to each word feature vector and all the neighboring word feature vectors of each word feature vector, the intermediate feature vector of each word is calculated. The intermediate feature vector of each word and the word feature vector of each word are added to obtain the word feature sum vector of each word. Then, the dimension transformation of the word feature sum vector of each word is performed to obtain the target feature vector of each word. Finally, the average value of all target feature vectors corresponding to each target text is calculated to obtain the text feature vector of each target text.After obtaining the visual feature vectors of each target object and the text feature vectors of each target text, the server first sums the visual and text feature vectors to obtain multiple summed feature vectors, and then multiplies the visual and text feature vectors to obtain multiple dot-product feature vectors. Next, it concatenates the corresponding summed feature vectors, dot-product feature vectors, visual feature vectors, and text feature vectors to obtain multiple fused feature vectors. Then, it calculates the matching degree between each visual and text feature vector based on these fused feature vectors, and identifies the target object with the highest matching degree among all fused feature vectors corresponding to each target text as the object localization result. After obtaining the object localization result, the server associates the target text corresponding to the object localization result with the video, using the target text as the video's title or description. When a user logs into a social media platform via a smartphone or computer, the server pushes the associated target text and video to the user through the social media platform.

[0176] Scene 2

[0177] The video object localization method provided in this invention can be applied to scenarios of text-based video search. Specifically, in response to detecting that a user searches for a video corresponding to a target title through a video sharing platform on a smartphone or computer, the server performs vector mapping on each word in the target title to obtain word vectors for each word, and performs feature encoding on each word vector to obtain word feature vectors for each word. Then, all word feature vectors are input into a pre-trained graph neural network model for feature extraction to obtain the local relation features of each word feature vector to each word feature vector. Based on all the local relation features corresponding to each word feature vector, the global relation features of all the neighboring word feature vectors to each word feature vector are calculated. Then, based on the global relation features corresponding to each word feature vector and all the neighboring word feature vectors of each word feature vector, the intermediate feature vector of each word is calculated. The intermediate feature vector of each word and the word feature vector of each word are added to obtain the word feature sum vector of each word. Then, the word feature sum vector of each word is dimensionally transformed to obtain the target feature vector of each word, and all target features are calculated. The average value of the feature vectors is used to obtain the text feature vector of the target title. While obtaining the text feature vector of the target title, the server simultaneously detects objects in each video frame of each video to obtain multiple object detection boxes. These object detection boxes are then aggregated based on the target object to obtain object information for each target object in each video. The multiple video frames corresponding to the object information of each target object are then divided based on the time dimension to obtain multiple video frame segments. In each video frame segment, the global and local spatiotemporal features of each target object are obtained. These global and local spatiotemporal features are then concatenated to obtain the concatenated feature vector of each target object in each video frame. After obtaining the concatenated feature vector of each target object in each video frame, the server calculates the average value of all concatenated feature vectors of each target object in each video frame segment to obtain the spatiotemporal feature vector of each target object in each video frame segment. Then, feature encoding is performed on each spatiotemporal feature vector corresponding to each target object to obtain the visual encoding vector of each target object in each video frame segment. Finally, the average value of all visual encoding vectors corresponding to each target object is calculated to obtain the visual feature vector of each target object in each video.After obtaining the visual feature vectors of each target object and the text feature vector of the target title, the server first sums the visual feature vectors and the text feature vector of the target title to obtain multiple summed feature vectors. Then, it performs dot product on each visual feature vector and the text feature vector of the target title to obtain multiple dot product feature vectors. Next, it concatenates the corresponding summed feature vectors, dot product feature vectors, visual feature vectors, and text feature vectors of the target title to obtain multiple fused feature vectors. Then, it calculates the matching degree between each visual feature vector and the text feature vector of the target title based on these fused feature vectors. Finally, it identifies the target object with the highest matching degree among all fused feature vectors corresponding to the target title as the object localization result. After obtaining the object localization result, the server pushes the video corresponding to the object localization result to the user through a video sharing platform.

[0178] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0179] Reference Figure 5 This invention also discloses a video object positioning device 500, which can implement the video object positioning method as described in the previous embodiments. The video object positioning device 500 includes:

[0180] The object information acquisition unit 510 is used to acquire object information of each target object in the video;

[0181] The first vector acquisition unit 520 is used to acquire the visual feature vectors of each target object based on the object information of each target object.

[0182] The second vector acquisition unit 530 is used to acquire multiple target texts, vectorize each word in each target text, and obtain the word feature vector of each word.

[0183] The third vector acquisition unit 540 is used to acquire the text feature vector of each target text based on the feature vectors of all words and the pre-trained graph neural network model.

[0184] The information fusion unit 550 is used to fuse various visual feature vectors and various text feature vectors to obtain multiple fused feature vectors.

[0185] The matching calculation unit 560 is used to obtain the matching degree between each visual feature vector and each text feature vector based on each fused feature vector.

[0186] The object localization unit 570 is used to determine the target object with the highest matching degree among all the fused feature vectors corresponding to each target text as the object localization result.

[0187] In one embodiment, the graph neural network model includes nodes corresponding to word feature vectors, and the relationships between nodes correspond to the relationships between words in the target text; the third vector acquisition unit 540 is further configured to:

[0188] All word feature vectors are input into a pre-trained graph neural network model for feature extraction, and the global relationship features of all neighboring word feature vectors to each word feature vector are obtained.

[0189] Based on the global relation features corresponding to each word's feature vector, and the feature vectors of all neighboring words of each word's feature vector, the target feature vector of each word is calculated.

[0190] Calculate the average value of all target feature vectors corresponding to each target text to obtain the text feature vector of each target text.

[0191] In one embodiment, the third vector acquisition unit 540 is further configured to:

[0192] All word feature vectors are input into a pre-trained graph neural network model for feature extraction, which yields the local relation features between each word feature vector and each adjacent word feature vector.

[0193] Based on all the local relation features corresponding to each word's feature vector, the global relation features of all neighboring word feature vectors to each word's feature vector are calculated.

[0194] In one embodiment, the third vector acquisition unit 540 is further configured to:

[0195] Based on the global relation features corresponding to each word's feature vector and the feature vectors of all neighboring words of each word's feature vector, the intermediate feature vector of each word is calculated.

[0196] The target feature vector of each word is calculated based on the intermediate feature vector and the word feature vector of each word.

[0197] In one embodiment, the third vector acquisition unit 540 is further configured to:

[0198] The intermediate feature vector of each word is added to the word feature vector of each word to obtain the word feature sum vector of each word;

[0199] The dimension transformation is performed on the sum of the word features of each word to obtain the target feature vector of each word.

[0200] In one embodiment, the second vector acquisition unit 530 is further configured to:

[0201] Vector mapping is performed on each word in each target text to obtain the word vector of each word;

[0202] The word vectors of each word are encoded to obtain the word feature vectors of each word.

[0203] In one embodiment, the object information of the target object corresponds to multiple video frames; the first vector acquisition unit 520 is further configured to:

[0204] The video frames corresponding to the object information of each target object are divided based on the time dimension to obtain multiple video frame segments.

[0205] Obtain the spatiotemporal feature vectors of each target object in each video frame segment;

[0206] Based on all the spatiotemporal feature vectors corresponding to each target object, the visual feature vector of each target object is calculated.

[0207] In one embodiment, the first vector acquisition unit 520 is further configured to:

[0208] In each video frame of each video frame segment, obtain the global and local spatiotemporal features of each target object;

[0209] Global and local spatiotemporal features are concatenated to obtain the concatenated feature vector of each target object in each video frame;

[0210] Calculate the average value of all spliced ​​feature vectors of each target object in each video frame segment to obtain the spatiotemporal feature vector of each target object in each video frame segment.

[0211] In one embodiment, the first vector acquisition unit 520 is further configured to:

[0212] Feature encoding is performed on the spatiotemporal feature vectors corresponding to each target object to obtain the visual encoding vectors of each target object in each video frame segment;

[0213] The average value of all visual encoding vectors corresponding to each target object is calculated to obtain the visual feature vector of each target object.

[0214] In one embodiment, the information fusion unit 550 is further configured to:

[0215] The visual feature vectors and the text feature vectors are summed to obtain multiple summed feature vectors;

[0216] The visual feature vectors and the text feature vectors are multiplied by a dot to obtain multiple dot-multiplied feature vectors.

[0217] The corresponding summation feature vectors, dot product feature vectors, visual feature vectors, and text feature vectors are concatenated to obtain multiple fused feature vectors.

[0218] In one embodiment, the video includes multiple video frames; the object information acquisition unit 510 is further configured to:

[0219] Each object in each video frame is detected, resulting in multiple object detection boxes;

[0220] Aggregate all object detection boxes based on the target object to obtain object information for each target object in the video.

[0221] It should be noted that since the video object positioning device 500 of this embodiment can implement the video object positioning method of the previous embodiment, the video object positioning device 500 of this embodiment and the video object positioning method of the previous embodiment have the same technical principle and the same beneficial effect. In order to avoid repetition, it will not be described again here.

[0222] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0223] Reference Figure 6 The present invention also discloses a video object positioning device, the video object positioning device 600 comprising:

[0224] At least one processor 601;

[0225] At least one memory 602 is used to store at least one program;

[0226] When at least one program is executed by at least one processor 601, the video object positioning method as described in any of the preceding embodiments is implemented.

[0227] This invention also discloses a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to implement the video object positioning method as described in any of the preceding embodiments.

[0228] This invention also discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the video object positioning method as described in any of the preceding embodiments.

[0229] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0230] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0231] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0232] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0235] The step numbers in the above method embodiments are set only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A video object localization method, characterized in that, Includes the following steps: Obtain object information for each target object in the video; The visual feature vectors of each target object are obtained based on the object information of each target object. Multiple target texts are obtained, and each word in each target text is vectorized to obtain the word feature vector of each word; Based on all the word feature vectors and the pre-trained graph neural network model, text feature vectors of each target text are obtained. The graph neural network model includes nodes corresponding to the word feature vectors, and the relationship between the nodes corresponds to the relationship between the words in the target text. The text feature vectors of the target text are used to associate each target text with all the word feature vectors through the graph neural network model. The graph neural network includes a graph attention network model, which connects multiple graph convolutional layers through a dense connection pattern. The visual feature vectors and the text feature vectors are fused to obtain multiple fused feature vectors. Based on each of the fused feature vectors, and using a fully connected network model to obtain the matching degree between each of the visual feature vectors and each of the text feature vectors, the fully connected network model is trained with ranking loss and mutual exclusion loss as training objectives; Among all the fused feature vectors corresponding to each target text, the target object corresponding to the one with the highest matching degree is determined as the object localization result.

2. The video object localization method according to claim 1, characterized in that, The step of obtaining the text feature vector of each of the target texts based on all the word feature vectors and the pre-trained graph neural network model includes: All the word feature vectors are input into the pre-trained graph neural network model for feature extraction, and the global relationship features of all neighboring word feature vectors to each word feature vector are obtained. Based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector, the target feature vector of each word is calculated. The average value of all the target feature vectors corresponding to each target text is calculated to obtain the text feature vector of each target text.

3. The video object localization method according to claim 2, characterized in that, The step of inputting all the word feature vectors into a pre-trained graph neural network model for feature extraction, and obtaining the global relationship features of all neighboring word feature vectors to each word feature vector, includes: All the word feature vectors are input into a pre-trained graph neural network model for feature extraction, to obtain the local relation features of each neighboring word feature vector to each word feature vector. Based on all the local relation features corresponding to each word feature vector, the global relation features of all the neighboring word feature vectors to each word feature vector are calculated.

4. The video object localization method according to claim 2, characterized in that, The step of calculating the target feature vector of each word based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector includes: Based on the global relation features corresponding to each word feature vector and the feature vectors of all neighboring words of each word feature vector, the intermediate feature vector of each word is calculated. The target feature vector of each word is calculated based on the intermediate feature vector of each word and the word feature vector of each word.

5. The video object localization method according to claim 4, characterized in that, The step of calculating the target feature vector of each word based on the intermediate feature vector of each word and the word feature vector of each word includes: The intermediate feature vector of each word and the word feature vector of each word are added together to obtain the word feature sum vector of each word; The dimension transformation is performed on the sum of the word features of each word to obtain the target feature vector of each word.

6. The video object localization method according to claim 1, characterized in that, The step of vectorizing each word in the target text to obtain the word feature vector of each word includes: Vector mapping is performed on each word in each of the target texts to obtain the word vectors of each word; The word vectors of each of the aforementioned words are feature-encoded to obtain the word feature vectors of each of the aforementioned words.

7. The video object localization method according to claim 1, characterized in that, The object information of the target object corresponds to multiple video frames; The step of obtaining the visual feature vector of each target object based on the object information of each target object includes: The video frames corresponding to the object information of each target object are divided based on the time dimension to obtain multiple video frame segments; Obtain the spatiotemporal feature vectors of each target object in each of the video frame segments; Based on all the spatiotemporal feature vectors corresponding to each target object, the visual feature vector of each target object is calculated.

8. The video object localization method according to claim 7, characterized in that, The step of obtaining the spatiotemporal feature vectors of each target object in each of the video frame segments includes: In each of the video frames of each video frame segment, the global spatiotemporal features and local spatiotemporal features of each target object are obtained; The global spatiotemporal features and the local spatiotemporal features are concatenated to obtain the concatenated feature vector of each target object in each video frame; The average value of all the spliced ​​feature vectors of each target object in each of the video frame segments is calculated to obtain the spatiotemporal feature vector of each target object in each of the video frame segments.

9. The video object localization method according to claim 7, characterized in that, The step of calculating the visual feature vector of each target object based on all the spatiotemporal feature vectors corresponding to each target object includes: Feature encoding is performed on each of the spatiotemporal feature vectors corresponding to each of the target objects to obtain the visual encoding vectors of each of the target objects in each of the video frame segments; The average value of all the visual encoding vectors corresponding to each target object is calculated to obtain the visual feature vector of each target object.

10. The video object localization method according to claim 1, characterized in that, The process of fusing information from each of the visual feature vectors and each of the text feature vectors to obtain multiple fused feature vectors includes: The visual feature vectors and the text feature vectors are summed to obtain multiple summed feature vectors; The visual feature vectors and the text feature vectors are multiplied by a dot to obtain multiple dot-multiplied feature vectors. The corresponding summation feature vectors, dot product feature vectors, visual feature vectors, and text feature vectors are concatenated to obtain multiple fused feature vectors.

11. The video object localization method according to claim 1, characterized in that, The video includes multiple video frames; obtaining object information for each target object in the video includes: Each object in each of the video frames is detected to obtain multiple object detection boxes; Aggregate all the object detection boxes based on the target object to obtain the object information of each target object in the video.

12. A video object positioning device, characterized in that, include: The object information acquisition unit is used to acquire object information of each target object in the video. The first vector acquisition unit is used to acquire the visual feature vector of each target object based on the object information of each target object; The second vector acquisition unit is used to acquire multiple target texts, vectorize each word in each target text, and obtain the word feature vector of each word. The third vector acquisition unit is used to acquire the text feature vector of each target text based on all the word feature vectors and the pre-trained graph neural network model. The graph neural network model includes nodes corresponding to the word feature vectors, and the relationship between the nodes corresponds to the relationship between the words in the target text. The text feature vector of the target text is used to associate each target text with all the word feature vectors through the graph neural network model. The graph neural network includes a graph attention network model, which connects multiple graph convolutional layers through a dense connection pattern. An information fusion unit is used to fuse the visual feature vectors and the text feature vectors to obtain multiple fused feature vectors. The matching calculation unit is used to obtain the matching degree between each of the visual feature vectors and each of the text feature vectors based on each of the fused feature vectors and using a fully connected network model. The fully connected network model is trained with ranking loss and mutual exclusion loss as training objectives. The object localization unit is used to determine the target object with the highest matching degree among all the fused feature vectors corresponding to each target text as the object localization result.

13. A video object positioning device, characterized in that, include: At least one processor; At least one memory for storing at least one program; The video object localization method as described in any one of claims 1 to 11 is implemented when at least one of the programs is executed by at least one of the processors.

14. A computer-readable storage medium, characterized in that, It stores a processor-executable program, which, when executed by a processor, is used to implement the video object localization method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium and executes the computer program or the computer instructions, causing the computer device to perform the video object localization method as described in any one of claims 1 to 11.

Images