Object detection method and related device
By calculating the pixel-level correlation between visual features and text features for multiple fusion, the problem of insufficient fusion of image features and text features in the prior art is solved, and the accuracy of object detection is improved.
Patent Information
- Application Number
- CN202410987177.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-07-23
Smart Images

Figure CN118675091B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an object detection method and related equipment. The related equipment can be applied to an object detection device, an electronic device, a computer program product and a computer-readable storage medium. Background Art
[0002] In recent years, with the rapid development of computer vision technology, multi-object tracking (MOT) technology has become more and more mature. In order to improve the accuracy of multi-object tracking, text instructions can often be introduced on the basis of multi-object tracking to guide object detection in video frames. Current object detection methods often use a cross-attention mechanism to fuse image features and text features, and perform object detection using the fused visual text features.
[0003] In the process of research and practice of current technology, the inventor of this application found that in the process of fusing image features and text features, only a simple cross-attention operation was performed, which made the fusion of image features and text features insufficient and relatively rough, thus resulting in low accuracy of object detection. Summary of the invention
[0004] The embodiments of the present application provide an object detection method and related equipment. The related equipment can apply for an object detection device, an electronic device, a computer program product and a computer-readable storage medium, which can improve the accuracy of object detection.
[0005] An object detection method, comprising:
[0006] Acquire a video to be detected and a detection text corresponding to the video to be detected, wherein the video to be detected includes at least one video frame to be detected;
[0007] Performing feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and performing feature extraction on the detected text to obtain text features;
[0008] Calculating a feature correlation between the visual feature and the text feature at each pixel position, wherein the feature correlation indicates a correlation degree between the visual feature and the text feature at the same pixel position;
[0009] Based on the feature correlation, the visual feature and the text feature are fused to obtain a visual text feature;
[0010] At least one target object is detected in the to-be-detected video frame according to the visual text feature and the visual feature.
[0011] Accordingly, an embodiment of the present application provides an object detection device, including:
[0012] An acquisition unit, used for acquiring a video to be detected and a detection text corresponding to the video to be detected, wherein the video to be detected includes at least one video frame to be detected;
[0013] An extraction unit, used to perform feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and perform feature extraction on the detected text to obtain text features;
[0014] A calculation unit, used for calculating the feature correlation between the visual feature and the text feature at each pixel position, wherein the feature correlation indicates the correlation degree between the visual feature and the text feature at the same pixel position;
[0015] A fusion unit, configured to fuse the visual feature with the text feature based on the feature correlation to obtain a visual text feature;
[0016] A detection unit is used to detect at least one target object in the video frame to be detected according to the visual text feature and the visual feature.
[0017] In some embodiments, the computing unit can be specifically used to map the visual features and the text features into the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features; extract the visual sub-features corresponding to each pixel position in the target visual features, and calculate the feature correlation between the visual sub-features and the target text features.
[0018] In some embodiments, the computing unit can be specifically used to identify the number of feature channels in the visual feature to obtain the number of visual channels, and identify the number of feature channels in the text feature to obtain the number of text channels; based on the number of visual channels and the number of text channels, determine the target number of channels in the feature space corresponding to each dimension, and fuse the text word features to obtain fused text features; adjust the number of channels of the visual feature and the number of channels of the fused text feature to the target number of channels, respectively, to obtain the target visual feature of each dimension and the target text feature corresponding to the target visual feature.
[0019] In some embodiments, the computing unit can be specifically used to extract features under each feature channel from the visual sub-features to obtain visual channel features; filter out features under the feature channel corresponding to the visual channel features from the target text features to obtain text channel features; and determine the feature correlation between the visual sub-features and the target text features based on the visual channel features and the text channel features.
[0020] In some embodiments, the computing unit can be specifically used to fuse the visual channel features with the text channel features to obtain fused channel features corresponding to each feature channel; accumulate the fused channel features at each pixel position to obtain a target fused feature corresponding to each pixel position; fuse the visual sub-features with the target text features to obtain a reference fused feature, and calculate the ratio between the reference fused feature and the target fused feature to obtain a feature correlation corresponding to each pixel position.
[0021] In some embodiments, the fusion unit can be specifically used to weight the visual sub-features based on the feature correlation to obtain the initial visual text features corresponding to each pixel position; and fuse the initial visual text features of the same dimension to obtain the visual text features of each dimension.
[0022] In some embodiments, the detection unit can be specifically used to sort the visual features based on the dimensions of the visual features, and filter out the visual features of the target dimension from the visual features based on the sorting results to obtain the current visual features; filter out the visual text features corresponding to the target dimension from the visual text features to obtain the current visual text features; extract object features from the video frame to be detected based on the current visual text features and the current visual features, and determine at least one target object in the video frame to be detected based on the object features.
[0023] In some embodiments, the detection unit can be specifically used to update the preset object features based on the current visual text features and the current visual features, and use the updated object features as the preset object features; return to execute the step of filtering out the visual features of the target dimension from the visual features based on the sorting results, until each visual feature is the current visual feature, to obtain the object features.
[0024] In some embodiments, the detection unit can be specifically used to update the preset object features based on the current visual text features and the current visual features, obtain the initial object features, and use the initial object features as the preset object features; return to execute the step of updating the preset object features based on the current visual text features and the current visual features until the preset number of updates is reached, and obtain the updated object features.
[0025] In some embodiments, the detection unit can be specifically used to perform self-attention weighting on the object sub-features to obtain candidate object features; perform cross-attention weighting on the candidate object features, current visual text features and current visual features to obtain current object features; and map the current object features to a preset object feature space to obtain initial object features.
[0026] In some embodiments, the extraction unit can be specifically used to perform multi-scale feature extraction on the video frame to be detected to obtain initial visual features of at least one feature scale; preprocess the initial visual features to obtain candidate visual features of at least one dimension; and perform feature interaction on the candidate visual features according to a preset interaction direction to obtain visual features of at least one dimension.
[0027] In some embodiments, the extraction unit can be specifically used to screen out the target initial visual features from the initial visual features based on the feature scale, and adjust the size of the target initial visual features to obtain adjusted visual features; splice the adjusted visual features with preset position features to obtain spliced visual features, and the spliced visual features include at least one visual sub-feature; perform self-attention weighting on the visual sub-features to obtain weighted visual features, and update the initial visual features based on the weighted visual features to obtain candidate visual features of at least one dimension.
[0028] In some embodiments, the extraction unit can be specifically used to sort the candidate visual features based on the feature sizes of the candidate visual features to obtain a first size sorting result; based on the first size sorting result, perform feature interaction on the candidate visual features in the first interaction direction to obtain at least one candidate interaction feature; based on the feature sizes of the candidate interaction features, sort the candidate interaction features to obtain a second size sorting result; based on the second size sorting result, perform feature interaction on the candidate interaction features in the second interaction direction to obtain at least one interaction feature, and use each interaction feature as a visual feature of one dimension.
[0029] In some embodiments, the extraction unit can be specifically used to screen out the candidate visual feature with the smallest feature size from the candidate visual features to obtain the current candidate visual feature, and use the current candidate visual feature as the first candidate interaction feature; based on the first size sorting result, screen out the next candidate visual feature of the current candidate visual feature from the candidate visual features to obtain the first feature to be interacted with of the current candidate visual feature; fuse the current candidate visual feature with the first feature to be interacted with to obtain the second candidate interaction feature, and use the first feature to be interacted with as the current candidate visual feature; return to execute the step of screening out the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result, until all candidate visual features complete feature interaction, and obtain at least one candidate interaction feature, and the candidate interaction features include the first candidate interaction feature and the second candidate interaction feature.
[0030] In some embodiments, the extraction unit can be specifically used to perform feature channel conversion on the current candidate visual feature to obtain a converted visual feature with a preset number of channels; amplify the feature size of the converted visual feature to obtain an enlarged visual feature, and the feature size of the enlarged visual feature is the same as the feature size of the first feature to be interacted with; perform feature channel conversion on the first feature to be interacted with to obtain a converted first feature to be interacted with with a preset number of channels; and splice the enlarged visual feature with the converted first feature to be interacted with to obtain a second candidate interactive feature.
[0031] In some embodiments, the extraction unit can be specifically used to screen out the candidate interaction feature with the largest feature size from the candidate interaction features to obtain the current interaction feature, and fuse the features of different feature channels in the current interaction feature to obtain the first interaction feature; based on the second size sorting result, screen out the next candidate interaction feature of the current interaction feature from the candidate interaction features to obtain the second feature to be interacted with of the current interaction feature; fuse the current interaction feature with the second feature to be interacted with to obtain the second interaction feature, and use the second feature to be interacted with as the current interaction feature; return to execute the step of screening out the next candidate interaction feature of the current interaction feature from the candidate interaction features based on the second size sorting result, until all candidate interaction features complete feature interaction, and obtain at least one interaction feature, and the interaction features include the first interaction feature and the second interaction feature.
[0032] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the object detection method provided in the embodiment of the present application.
[0033] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any object detection method provided in the embodiment of the present application.
[0034] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the object detection method provided in the embodiment of the present application.
[0035] After obtaining the video to be detected and the detection text corresponding to the video to be detected, the embodiment of the present application performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, and then calculates the feature correlation between the visual features and the text features at each pixel position, and based on the feature correlation, fuses the visual features with the text features to obtain visual text features, and then detects at least one target object in the video frame to be detected based on the visual text features and the visual features; because this scheme can calculate the feature correlation between the visual features and the text features at each pixel position after extracting the video features and the text features, and fuse the visual features with the text features through the feature correlation, and adopts a pixel-level image-text feature fusion method in the fusion process, thereby realizing fine-grained fusion between visual features and text features, and therefore, can improve the accuracy of object detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0037] Figure 1 is a scene schematic diagram of the object detection method provided in an embodiment of the present application;
[0038] Figure 2 is a flow chart of an object detection method provided in an embodiment of the present application;
[0039] Figure 3 is a schematic diagram of multi-scale visual feature extraction provided by an embodiment of the present application;
[0040] Figure 4 It is a schematic diagram of the fusion of visual features and text features provided in the embodiment of the present application;
[0041] Figure 5 is a schematic diagram of extracting object features from a video frame to be detected provided by an embodiment of the present application;
[0042] Figure 6 It is a schematic diagram of the overall framework of object detection provided by an embodiment of the present application;
[0043] Figure 7 is another flow chart of the object detection method provided in an embodiment of the present application;
[0044] Figure 8 is a schematic diagram of the structure of an object detection device provided in an embodiment of the present application;
[0045] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0047] The embodiment of the present application provides an object detection method and related equipment, which may include an object detection device, an electronic device, a computer program product, and a computer-readable storage medium. The object detection device may be integrated in an electronic device, which may be a server or a terminal.
[0048] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.
[0049] For example, see Figure 1Taking the object detection device integrated in an electronic device as an example, after obtaining the video to be detected and the detection text corresponding to the video to be detected, the electronic device performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, then calculates the feature correlation between the visual features and the text features at each pixel position, and based on the feature correlation, fuses the visual features with the text features to obtain visual text features, and then detects at least one target object in the video frame to be detected based on the visual text features and the visual features, thereby improving the accuracy of object detection.
[0050] Among them, the object detection method provided in the embodiment of the present application involves computer vision technology (CV) and natural language processing (NLP) in artificial intelligence (AI). In the embodiment of the present application, visual features can be extracted from the video frame to be detected, and text features can be extracted from the detected text. Then, the feature correlation between the visual features and the text features at each pixel position is calculated. Based on the feature correlation, the visual features and the text features are fused to obtain visual text features. Then, based on the visual text features and the visual features, at least one target object is detected in the video frame to be detected, thereby improving the accuracy of object detection.
[0051] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0052] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics and other technologies. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0053] Among them, computer vision technology (Computer Vision, CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, detect and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, MAE, etc. can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0054] Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing involves natural language, that is, the language people use in daily life, which is closely related to linguistics research; it also involves computer science and mathematics. It is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to linguistics research. The important technology for model training in the field of artificial intelligence, the pre-training model, is developed from the large language model in the field of NLP. After fine-tuning, the large language model can be widely used in downstream tasks. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.
[0055] It can be understood that in the specific implementation of the present application, related data such as the video to be detected and the detection text of the object are involved. When the following embodiments of the present application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0056] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0057] This embodiment will be described from the perspective of an object detection device, which can be specifically integrated into an electronic device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a wearable device, a virtual reality device, or other smart devices that can perform object detection.
[0058] An object detection method, comprising:
[0059] A video to be detected and a detection text corresponding to the video to be detected are obtained, wherein the video to be detected includes at least one frame of the video to be detected, feature extraction is performed on the video frame to be detected to obtain visual features of at least one dimension, and feature extraction is performed on the detection text to obtain text features, and feature correlations between the visual features and the text features at each pixel position are calculated, wherein the feature correlation indicates the degree of correlation between the visual features and the text features at the same pixel position, and based on the feature correlation, the visual features are fused with the text features to obtain visual text features, and at least one target object is detected in the video frame to be detected according to the visual text features and the visual features.
[0060] like Figure 2 As shown, the specific process of the object detection method is as follows:
[0061] 101. Obtain a video to be detected and a detection text corresponding to the video to be detected.
[0062] The video to be detected includes at least one frame of video to be detected. The video to be detected can be understood as a video that needs to be subjected to object detection or multi-target tracking. Multi-object tracking (MOT) is an important task in computer vision, which aims to track multiple targets simultaneously in a video sequence. First, all targets need to be detected in each video frame, and the detected targets are associated between different frames to determine which detections belong to the same target and are assigned a unique ID (object identifier). The output result is the bounding box, category and ID of the tracked target in the corresponding frame.
[0063] The detection text may include text instructions for guiding object detection in the video to be detected, and the detection text may indicate an object to be detected or a target to be tracked. For example, if the detection text is a black dog, it may indicate detecting or tracking a black dog in the video to be detected.
[0064] There are many ways to obtain the video to be detected and the detection text corresponding to the video to be detected, which can be as follows:
[0065] For example, a video to be detected and a detection text corresponding to the video to be detected uploaded by a terminal or a client can be received, or at least one original video can be obtained from a video platform or a network as the video to be detected, and the detection text corresponding to the video to be detected can be screened out from a text collection, or, based on a target detection task, at least one detection text can be screened out from a text collection to obtain a text collection, and at least one video to be detected corresponding to the target detection task can be screened out from a content platform, and the detection text corresponding to each video to be detected can be determined in the text collection, or, when the number of videos to be detected and the detection text exceeds a preset number threshold, an object detection request can also be received, and the object detection request carries the storage address of the video to be detected and the detection text corresponding to the video to be detected, and based on the storage address, the video to be detected and the detection text corresponding to the video to be detected are obtained, and so on.
[0066] 102. Perform feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and perform feature extraction on the detected text to obtain text features.
[0067] There are many ways to extract features from the video frame to be detected, which can be as follows:
[0068] For example, multi-scale feature extraction may be performed on the video frame to be detected to obtain initial visual features of at least one feature scale, the initial visual features may be preprocessed to obtain candidate visual features of at least one dimension, and feature interaction may be performed on the candidate visual features according to a preset interaction direction to obtain visual features of at least one dimension, which may be specifically as follows:
[0069] (1) Perform multi-scale feature extraction on the video frame to be detected to obtain initial visual features of at least one feature scale.
[0070] Among them, the feature scale can be understood as the output corresponding to different network layers in the extraction network. The deeper the feature scale, the higher the level of semantic information contained in the corresponding visual feature. Conversely, the shallower the feature scale, the lower the level of semantic information contained in the corresponding visual feature. The feature sizes corresponding to different feature scales are also different. The deeper the feature scale, the smaller the corresponding feature size. The deeper the network layer in the extraction network, the deeper the feature scale of the output visual feature. For example, taking the visual feature output of the third-to-last network layer in the extraction network as F3, the visual feature output of the second-to-last network layer as F4, and the visual feature output of the last network layer as F5 as an example, the visual feature F5 has the deepest feature scale, the smallest corresponding feature size, and the highest level of semantic information. Conversely, the visual feature F3 has the shallowest feature scale, the largest corresponding feature size, and the lowest level of semantic information.
[0071] There are many ways to extract multi-scale features from the video frame to be detected, which can be as follows:
[0072] For example, a visual feature extraction network of an object detection model may be used to perform multi-scale feature extraction on a video frame to be detected, and visual features output by at least one preset network layer in the visual feature extraction network may be used as initial visual features of the corresponding feature scale.
[0073] Among them, there can be many types of visual feature extraction networks, for example, they can include the RN50 version of CLIP (a multimodal pre-trained neural network) image encoder, LIip (a feature encoding network) or other networks that can perform multi-scale feature extraction on video frames, etc. The visual feature extraction network can include multiple network layers. The preset network layer can be understood as a network layer that outputs visual features in advance, for example, it can be set to the last three layers or any number / position of network layers. It should be noted that the visual features output by each preset network layer can be the initial visual features of a feature scale.
[0074] (2) Preprocessing the initial visual features to obtain candidate visual features of at least one dimension.
[0075] For example, based on the feature scale, the target initial visual feature can be screened out from the initial visual features, and the size of the target initial visual feature can be adjusted to obtain the adjusted visual feature, the adjusted visual feature can be spliced with the preset position feature to obtain the spliced visual feature, the spliced verse feature includes at least one visual sub-feature, the visual sub-feature is self-attention weighted to obtain the weighted visual feature, and the initial visual feature is updated based on the weighted visual feature to obtain the candidate visual feature of at least one dimension.
[0076] There are many ways to filter out target visual features from the initial visual features based on the feature scale. For example, the initial visual features with the deepest feature scale can be filtered out from the initial visual features to obtain the target visual features. Alternatively, the initial visual features can be sorted based on the feature scale, and based on the sorting results, the initial visual features with the deepest feature scale can be filtered out from the initial visual features to obtain the target visual features, and so on.
[0077] After the target visual features are screened out from the initial visual features based on the feature scale, the size of the target initial visual features can be adjusted to obtain the adjusted visual features. There are many ways to adjust the size of the target visual features. For example, the flatten layer (a data conversion network layer) in the feature extraction network can be used to flatten the target initial visual features to obtain the adjusted visual features.
[0078] After adjusting the initial visual feature of the target, the adjusted visual feature can be spliced with the preset position feature to obtain a spliced visual feature. The preset position feature can be a sine-cosine position code (feature). The spliced visual feature includes at least one visual sub-feature.
[0079] After the adjusted visual feature is concatenated with the preset position feature, the visual sub-features of the concatenated visual feature can be self-attention weighted to obtain the weighted visual feature. Self-attention weighting of the visual sub-features can enhance their feature representation, thereby capturing long-range dependencies in the features.
[0080] After self-attention weighting of the visual sub-features, the initial visual features can be updated based on the weighted visual features to obtain candidate visual features of at least one dimension. There are many ways to update the initial visual features. For example, the target initial visual features in the initial visual features can be replaced with the weighted visual features to obtain candidate visual features of at least one dimension.
[0081] (3) According to the preset interaction direction, feature interaction is performed on the candidate visual features to obtain visual features of at least one dimension.
[0082] The preset interaction direction may include a first interaction direction and a second interaction direction, the first interaction direction may be a self-directed interaction direction (path), and the second interaction direction may be a bottom-up interaction direction (path). Feature interaction in the first interaction direction may fuse high-level semantic information in deep candidate visual features into shallow candidate visual features. Feature interaction in the second interaction direction may fuse low-level semantic information in shallow candidate visual features into deep candidate visual features.
[0083] There are multiple ways to perform feature interaction on candidate visual features according to the preset interaction direction, which can be as follows:
[0084] For example, the candidate visual features can be sorted based on their feature sizes to obtain a first size sorting result; based on the first size sorting result, feature interaction can be performed on the candidate visual features in a first interaction direction to obtain at least one candidate interaction feature; based on their feature sizes, the candidate interaction features can be sorted to obtain a second size sorting result; based on the second size sorting result, feature interaction can be performed on the candidate interaction features in a second interaction direction to obtain at least one interaction feature; and each interaction feature can be regarded as a visual feature of one dimension.
[0085] Among them, according to the first size sorting result, there can be multiple ways to perform feature interaction on the candidate visual features according to the first interaction direction. For example, the candidate visual feature with the smallest feature size can be screened out from the candidate visual features to obtain the current candidate visual feature, and the current candidate visual feature is used as the first candidate interaction feature. Based on the first size sorting result, the next candidate visual feature of the current candidate visual feature is screened out from the candidate visual features to obtain the first feature to be interacted with the current candidate visual feature. The current candidate visual feature is fused with the first feature to be interacted with to obtain the second candidate interaction feature, and the first feature to be interacted with is used as the current candidate visual feature. Return to execute the step of screening out the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result, until all candidate visual features complete feature interaction, and obtain at least one candidate interaction feature, which includes the first candidate interaction feature and the second candidate interaction feature.
[0086] Among them, based on the first size sorting result, the next candidate visual feature of the current candidate visual feature is screened out from the candidate visual features. The next candidate visual feature here can be understood as the candidate visual feature that is ranked next to the current candidate visual feature in the first size sorting result sorted from small to large according to the feature size. For example, taking the current visual feature as the candidate visual feature with the smallest feature size, the next candidate visual feature can be the candidate visual feature with the second smallest feature size.
[0087] After the next candidate visual feature of the current candidate visual feature is screened out, the screened out first feature to be interacted with the current candidate visual feature can be fused to obtain a second candidate interactive feature. There are many ways to fuse the current candidate visual feature with the feature to be interacted. For example, the current candidate visual feature can be subjected to feature channel conversion to obtain a converted visual feature with a preset number of channels, the feature size of the converted visual feature can be enlarged to obtain an enlarged visual feature, the feature size of the enlarged visual feature is the same as the feature size of the first feature to be interacted with, the first feature to be interacted with is subjected to feature channel conversion to obtain a converted first feature to be interacted with with a preset number of channels, and the enlarged visual feature is spliced with the converted first feature to be interacted with to obtain a second candidate interactive feature.
[0088] There are many ways to convert the feature channels of the current candidate visual feature. For example, a 1*1 convolution can be used to convert the feature channel number of the current candidate visual feature into a preset channel number, thereby obtaining the converted visual feature. The preset channel number can be a preset channel number, for example, it can include 128, 256 or other values, and so on.
[0089] After the feature channel conversion is performed on the current candidate visual feature, the feature size of the converted visual feature can be enlarged to obtain the enlarged visual feature. There are many ways to enlarge the feature size of the converted visual feature. For example, the target feature size of the first feature to be interacted with can be obtained, and based on the target feature size, the feature size of the converted visual feature can be interpolated to the same as the first feature to be interacted with by using a bilinear interpolation method, thereby obtaining the enlarged visual feature.
[0090] Among them, the method of performing feature channel conversion on the first feature to be interacted with can be similar to the method of performing feature channel conversion on the current candidate visual feature, as described above, and will not be repeated here.
[0091] After the first feature to be interacted with is converted into a feature channel, the enlarged visual feature can be spliced with the converted first feature to be interacted with, so as to obtain a second candidate interaction feature. There are many ways to splice the enlarged visual feature with the converted first feature to be interacted with, for example, the enlarged visual feature can be spliced with the converted first feature to be interacted with in the dimension of the feature channel, so as to obtain the second candidate interaction feature.
[0092] After the current candidate visual feature is fused with the first feature to be interacted, the first feature to be interacted can be used as the current candidate visual feature, and then the step of selecting the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result is returned until the candidate visual features complete feature interaction, thereby obtaining at least one candidate interaction feature. The candidate interaction feature may include the first candidate interaction feature and the second candidate interaction feature.
[0093] After the candidate visual features are subjected to feature interaction according to the first interaction direction according to the first size sorting result, the candidate interaction features can be sorted based on the feature size of at least one candidate interaction feature to obtain a second size sorting result. There are many ways to sort the candidate interaction features. For example, the candidate interaction features can be sorted from large to small according to the feature size to obtain a second size sorting result.
[0094] After the candidate interaction features are sorted, the candidate interaction features can be subjected to feature interaction in the second interaction direction according to the second size sorting result to obtain at least one interaction feature. There can be multiple ways to perform feature interaction on the candidate interaction features in the second interaction direction. For example, the candidate interaction feature with the largest size can be screened out from the candidate interaction features to obtain the current interaction feature, and the features of different feature channels in the current interaction feature can be fused to obtain the first interaction feature. Based on the second size sorting result, the next candidate interaction feature of the current interaction feature can be screened out from the candidate interaction features to obtain the second feature to be interacted with the current interaction feature, the current interaction feature can be fused with the second feature to be interacted with to obtain the second interaction feature, and the second feature to be interacted with can be used as the current interaction feature. Return to execute the step of screening out the next candidate interaction feature of the current interaction feature from the candidate interaction features based on the second size sorting result, until all candidate interaction features complete feature interaction, and obtain at least one interaction feature, which includes the first interaction feature and the second interaction feature.
[0095] Among them, there are many ways to fuse the current interaction feature with the second feature to be interacted. For example, the current interaction feature can be converted into a feature channel to obtain a converted interaction feature with a preset number of channels, the feature size of the converted interaction feature can be reduced to obtain an adjusted interaction feature, the second feature to be interacted can be converted into a feature channel to obtain a converted second feature to be interacted with a preset number of channels, and the adjusted interaction feature can be spliced with the converted second feature to be interacted with to obtain a second interaction feature.
[0096] Among them, the method of converting the feature channel of the current interactive feature can be similar to the method of converting the number of channels of the current candidate visual feature. Please see the above description for details, and will not be repeated here.
[0097] After performing feature channel conversion on the current interactive feature, the feature size of the converted visual feature can be reduced. There are many ways to reduce the size. For example, the feature size of the converted visual feature can be halved to obtain the adjusted interactive feature.
[0098] Among them, the method of converting the feature channel of the first feature to be interacted can be similar to the method of converting the number of channels of the current candidate visual feature. Please see the above description for details, and will not be repeated here.
[0099] After the feature channel interaction is performed on the first feature to be interacted, the adjusted interactive feature can be spliced with the converted second feature to be interacted, thereby obtaining the second interactive feature. The manner of splicing the adjusted interactive feature with the converted second feature to be interacted can be similar to the manner of splicing the magnified visual feature with the converted first feature to be interacted, as described above, and will not be described in detail here.
[0100] After the current interaction feature is merged with the second feature to be interacted, the second feature to be interacted can be used as the current interaction feature. Then, the step of selecting the next candidate interaction feature of the current interaction feature from the candidate interaction features based on the second size sorting result is returned until all candidate interaction features complete feature interaction, and at least one interaction feature is obtained. The interaction feature may include the first interaction feature and the second interaction feature.
[0101] After performing feature interaction on the candidate interaction features according to the second interaction direction based on the sorting result of the second dimension, each interaction feature can be regarded as a visual feature of one dimension.
[0102] Among them, in this scheme, in the process of visual feature extraction, by performing self-attention operation on the deepest feature map (target initial visual feature), and then using the pyramid structure, the candidate visual features of different scales are subjected to feature interaction in two interactive directions (paths) from top to bottom and from bottom to top, so that multi-scale visual features can be fully integrated. Taking the extraction of initial visual features including initial visual features F3, F4 and F5 of three feature scales, and the preset number of channels is 128 as an example, the extraction process can be as follows Figure 3 As shown, the specific details can be as follows:
[0103] (1) First, use flatten to flatten the deepest target initial visual feature F5, then add sine and cosine position encoding, and then input it into the encoder for self-attention operation, and finally output the feature map after the attention operation, denoted as A5.
[0104] (2) Top-down direction (path)
[0105] a) Use 1x1 convolution to convert the number of channels of A5 to 128, then use bilinear interpolation to interpolate the width and height of A5 to the same as F4, then use 1x1 convolution to convert the number of channels of F4 to 128, and then splice the two processed feature maps in the channel dimension to obtain A4 with 256 channels.
[0106] b) Use 1x1 convolution to convert the number of channels of A4 to 128, then use bilinear interpolation to interpolate the width and height of A4 to the same as F3, then use 1x1 convolution to convert the number of channels of F3 to 128, and then splice the two processed feature maps in the channel dimension to obtain A3 with 256 channels.
[0107] (3) Bottom-down direction (path)
[0108] a) Use 1x1 convolution to process A3 and fuse the information of different channels to obtain F3'.
[0109] b) Use 1x1 convolution to convert the number of channels of F3' to 128, then use 3x3 convolution to halve the width and height of F3', then use 1x1 convolution to convert the number of channels of A4 to 128, and then splice the two processed feature maps in the channel dimension to obtain F4' with a channel number of 256.
[0110] c) Use 1x1 convolution to convert the number of channels of F4' to 128, then use 3x3 convolution to halve the width and height of F4', then use 1x1 convolution to convert the number of channels of A5 to 128, and then splice the two processed feature maps in the channel dimension to obtain F5' with a channel number of 256.
[0111] F3', F4' and F5' are used as visual features of multiple dimensions extracted
[0112] There are many ways to extract features from the detected text. For example, a text feature extraction network can be used to extract an embedded representation (text embedding) from the detected text to obtain text features.
[0113] There are many types of text feature extraction networks, for example, CLIP's text transformer (a text feature extraction network) or other networks that can extract text features, and so on.
[0114] 103. Calculate the feature correlation between the visual feature and the text feature at each pixel position.
[0115] Among them, the feature correlation can indicate the correlation degree between the visual feature and the text feature at the same pixel position.
[0116] There are many ways to calculate the feature correlation between the visual feature and the text feature at each pixel position, which can be as follows:
[0117] For example, the visual features and text features can be mapped to the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features. The visual sub-features corresponding to each pixel position are extracted from the target visual features, and the feature correlation between the visual sub-features and the target text features is calculated. Specifically, it can be as follows:
[0118] S1. Map the visual features and text features into the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features.
[0119] The text feature includes at least one text word feature. There are many ways to map the visual feature and the text feature into the same feature space, which can be as follows:
[0120] For example, the number of feature channels can be identified in the visual features to obtain the number of visual channels, and the number of feature channels can be identified in the text features to obtain the number of text channels. Based on the number of visual channels and the number of text channels, the target number of channels in the feature space corresponding to each dimension is determined, and the text word features are fused to obtain fused text features. The number of channels of the visual features and the number of channels of the fused text features are adjusted to the target number of channels, respectively, to obtain the target visual features of each dimension and the target text features corresponding to the target visual features.
[0121] Among them, in the process of mapping visual features and text features to the same feature space, the dimension of visual features is [b, c1, w, h], and the dimension of text features is [b, p, c2], where b refers to the batch size, c1 and c2 represent the number of channels of visual features and text features respectively, w and h are the width and height of visual features respectively, and p refers to the number of tokens of text features. In the process of feature mapping, the visual feature dimension output after the visual feature passes through the Project Layer (a network layer) is [b, w, h, k], and the feature dimension output after the text feature passes through the Project Layer is [b, 1, k], that is, the text feature in the Project Layer merges the token embedding (text word feature) of each word into the sentence embedding (fused text feature) of the entire sentence.
[0122] S2. Extract the visual sub-features corresponding to each pixel position from the target visual features, and calculate the feature correlation between the visual sub-features and the target text features.
[0123] For example, the pixel position of each pixel in the video frame to be detected is obtained, and the visual sub-feature corresponding to each pixel position is extracted from the target visual feature.
[0124] After extracting the visual sub-features corresponding to each pixel position from the target visual features, the feature correlation between the visual sub-features and the target text features can be calculated. There are many ways to calculate the feature correlation between the visual sub-features and the target text features. For example, features under each feature channel can be extracted from the visual sub-features to obtain visual channel features, and features under the feature channel corresponding to the visual channel features can be screened out from the target text features to obtain text channel features. Based on the visual channel features and the text channel features, the feature correlation between the visual sub-features and the target text features can be determined.
[0125] There are many ways to determine the feature correlation between the visual sub-features and the target text features based on the visual channel features and the text channel features. For example, the visual channel features and the text channel features can be fused to obtain the fused channel features corresponding to each feature channel, the fused channel features at each pixel position are accumulated to obtain the target fused features corresponding to each pixel position, the visual sub-features and the target text features are fused to obtain the reference fused features, and the ratio between the reference fused features and the target fused features is calculated to obtain the feature correlation corresponding to each pixel position. Specifically, it can be shown in formula (1), which can be as follows:
[0126]
[0127] in, is the feature correlation at each pixel position, is the visual channel feature at the pixel position, is the text channel feature, and are the horizontal and vertical coordinates of the pixel position respectively. is the visual sub-feature corresponding to the pixel position, is the target text feature.
[0128] 104. Based on feature relevance, the visual features and text features are fused to obtain visual text features.
[0129] Among them, visual text features can be understood as features that integrate vision and text.
[0130] There are many ways to fuse visual features with text features based on feature relevance, which can be as follows:
[0131] For example, the visual sub-features may be weighted based on feature relevance to obtain the initial visual text features corresponding to each pixel position, and the initial visual text features of the same dimension may be fused to obtain the visual text features of each dimension.
[0132] There are many ways to weight the visual sub-features based on the feature relevance. For example, the target visual sub-features corresponding to the feature relevance are selected from the visual sub-features, and the target visual sub-features are weighted based on the feature relevance, so as to obtain the initial visual text features corresponding to each pixel position, which can be shown in formula (2). Specifically, it can be as follows:
[0133]
[0134] in, and are the horizontal and vertical coordinates of the pixel position respectively. is the initial visual text feature at the pixel position, The visual sub-feature corresponding to the pixel position, is the feature correlation corresponding to the pixel position.
[0135] After the initial visual sub-features are weighted based on feature relevance, the weighted initial visual text features of the same dimension can be fused to obtain the visual text features of each dimension.
[0136] In the fusion process of visual features and text features, this scheme first calculates the correlation between visual features and text features at the pixel level (i.e., feature correlation), and then uses the feature correlation as the weight of the visual feature to weight the visual feature, thereby obtaining the visual text features of each dimension. Specifically, Figure 4 shown.
[0137] 105. Detect at least one target object in the video frame to be detected according to the visual text features and the visual features.
[0138] For example, the visual features can be sorted based on their dimensions, and based on the sorting results, the visual features of the target dimension can be screened out from the visual features to obtain the current visual features, and the visual text features corresponding to the target dimension can be screened out from the visual text features to obtain the current visual text features, and object features can be extracted from the video frame to be detected based on the current visual text features and the current visual features, and based on the object features, at least one target object in the video frame to be detected can be determined.
[0139] There are many ways to filter out visual features of the target dimension from the visual features based on the sorting results. For example, based on the sorting results, the visual features with the deepest feature scale can be filtered out from the visual features as the current visual features corresponding to the target dimension. Alternatively, based on the sorting results, the visual features with the smallest feature size can be filtered out from the visual features as the current visual features corresponding to the target dimension, and so on.
[0140] After the visual features of the target dimension are screened out from the visual features based on the sorting results, the visual text features corresponding to the target dimension can be screened out from the visual text features. Then, object features are extracted from the video frame to be detected based on the current visual text features and the current visual features. The so-called object features can be understood as features indicating information such as the position, category, and identification of the target object contained in the video frame to be detected. There are many ways to extract object features from the video frame to be detected based on the current visual text features and the current visual features. For example, the preset object features can be updated based on the current visual text features and the current visual features, and the updated object features are used as the preset object features, and the step of screening out the visual features of the target dimension from the visual features based on the sorting results is returned until each visual feature is the current visual feature, and the object features are obtained.
[0141] The preset object feature can be understood as the initialized object feature (query, query∈R300×512), which is a vector that can be learned and used to represent the target object to be detected or tracked. The preset object feature includes at least one object sub-feature. Based on the current visual text feature and the current visual feature, there are many ways to update the preset object feature. For example, the object sub-feature can be self-attention weighted to obtain the candidate object feature, the candidate object feature, the current visual text feature and the current visual feature can be cross-attention weighted to obtain the current object feature, and the current object feature can be mapped to the preset object feature space to obtain the initial object feature.
[0142] There are many ways to perform cross-attention weighting on the candidate object features, the current visual text features and the current visual features. For example, the candidate object features can be used as query features (q), the current visual features can be used as key features (k), and the current visual text features can be used as value features (v). The query features, key features and value features can be cross-attention weighted to obtain the current object features.
[0143] After cross-attention weighting of the candidate object features, the current visual text features, and the current visual features, the weighted current object features can be mapped to the preset object feature space to obtain the initial object features. There are many ways to map the current object features to the preset object feature space. For example, a feedforward neural network can be used to map the current object features to the preset object feature space to obtain the initial object features.
[0144] After updating the preset object features based on the current visual features and the current visual features, the updated initial object features can be used as the preset object features. Then, the step of updating the preset object features based on the current visual text features and the current visual features is returned to execute until the preset number of updates is reached to obtain the updated object features.
[0145] The preset update times may be the preset number of times that the same pair of features (the current visual text feature and the corresponding current visual feature) updates the preset object feature, for example, once, twice, or multiple times.
[0146] After the preset object feature is updated based on the current visual text feature and the current visual feature, the updated object feature can be used as the preset object feature. Then, the step of filtering out the visual feature of the target dimension from the visual features based on the sorting result is returned until each visual feature is the current visual feature, thereby obtaining the object feature.
[0147] Among them, taking the visual features including F3', F4' and F5', the feature sizes of these three visual features decrease in sequence, the corresponding visual text features are H3, H4 and H5, and the preset update number is 1 as an example, the process of extracting the object feature (query) in the video frame to be detected in this scheme can be as follows Figure 5 As shown, first initialize the query (preset object features), then input F5', H5 and the initialized query into the first feature decoding network (Decoder Layer1), based on F5' and H5, update the initialized query through the first feature decoding network, then input the updated query, F4' and H4 into the second feature decoding network (Decoder Layer2), continue to update the updated query through the second feature decoding, and so on, until the initialized query is updated through the three-layer feature decoding network, thereby obtaining the object features of the video frame to be detected. Through three refinements, the object features can more accurately include the position, category and identification (ID) of the target object to be detected.
[0148] After extracting object features from the video frame to be detected based on the current visual text features and the current visual features, at least one target object in the video frame to be detected can be determined based on the object features. There are many ways to determine at least one target object in the video frame to be detected based on the object features. For example, the object features can be linearly processed to obtain the object confidence and object attribute information of at least one candidate object in the video frame to be detected, and at least one target object is screened out from the candidate objects based on the object confidence, and the target object attribute information of the target object is determined based on the object attribute information.
[0149] There are many ways to perform linear processing on object features. For example, an object prediction network (predict head) may be used to perform linear processing on object features, thereby obtaining the confidence and object attribute information of at least one candidate object.
[0150] After linear processing of the object features, at least one target object can be selected from the at least one candidate object obtained. There are many ways to select the target object, for example, at least one candidate object whose object confidence exceeds a preset confidence threshold can be selected from the candidate objects as the target object.
[0151] After the target object is screened out, the target object attribute information of the target object can be determined based on the object attribute information. The object attribute information may include object location, object category, object identification or other attribute information, etc. There may be multiple ways to determine the target object attribute information of the target object. For example, the object attribute information corresponding to the target object may be screened out from the object attribute information to obtain the target object attribute information.
[0152] Optionally, after determining at least one target object in the video frame to be detected, the video to be detected can be processed based on the target object attribute information of the target object. There can be multiple ways to process the video to be detected, for example, based on the target object attribute information, the target video frame can be screened out from the video frame to be detected of the video to be detected, and at least one target area can be identified in the target video frame, and the content of the target area can be edited, or, based on the target object attribute information, at least one target area can be identified in the video frame to be detected of the video to be detected, the target area can be rewritten, and according to the timing of the target area, the rewriting consistency of the content across frames can be achieved, and so on.
[0153] Among them, the target object identified in this scheme can be used in image (video) content editing or rewriting. For example, in the process of video editing, production or modification, the object appearing in the video image can be detected, and the object can be a specific content such as a specific scenery, and the area where the object is located can be located. The subsequent steps only need to edit, produce or modify the content of the area, thereby improving the processing efficiency of the video image or specific objects in the video image. Alternatively, the target that needs to be rewritten in the video frame can be tracked and located, and the timing consistency of the target that needs to be rewritten between video frames can be ensured. The located target area is rewritten, and according to the timing of the target area, the consistency of cross-frame content rewriting is achieved, thereby improving the efficiency of video content rewriting and content consistency.
[0154] Among them, when this scheme and other object detection methods (TransRMOT, iKUN) are respectively applied to the pre-step of video content rewriting in the downstream task - specific target detection, the experimental results can be shown in Table 1. From the results, it can be seen that this scheme is higher than other schemes in four experimental indicators (HOTA (an indicator of high-order tracking accuracy), DetA (an indicator for measuring association accuracy), AssA (an indicator for measuring detection accuracy) and MOTA (an indicator for the overall performance of detection and tracking)), that is, this scheme has better performance in detection positioning, as well as tracking association, which proves that this scheme can fully mine visual features and integrate visual and text features, thereby improving the accuracy of object detection.
[0155] Table 1
[0156]
[0157] In the process of object detection in the video to be detected, taking the extracted initial visual features F3 / F4 / F5 as an example, the overall object detection framework can be as follows: Figure 6 As shown in the figure, it can mainly include feature extraction (encoding) network (Backbone), visual feature fusion encoding network (FPN Encoder), image and text feature fusion network (FusionModule), feature decoding network (Refining-Decoder) and prediction network (Predict Head). The whole object detection process can be specifically as follows:
[0158] (1) Obtain video sequences (video to be detected) and text instructions (text to be detected).
[0159] (2) Use the image feature extraction network (img encoder) in Backbone to process the video sequence and extract three layers of feature maps (initial visual features) of different scales, denoted as F3 / F4 / F5. Use the text feature extraction network (text encoder) in the feature extraction network to process the text instructions and extract the embedded representation (textembedding) of the text instructions, denoted as Temd, to obtain the text features.
[0160] (3) The three layers of feature maps F3 / F4 / F5 of different scales are input into the FPN Encoder for attention operation and feature map fusion, and the fused visual feature map (visual feature) is output, which is represented as F3' / F4' / F5'.
[0161] (4) The fused visual feature map F3' / F4' / F5' and the text instruction embedding feature Temd are input into the FusionModule for image-text feature fusion, and the fused image-text features (visual text features) are output, which are represented as H3 / H4 / H5.
[0162] (5) The visual features F3' / F4' / F5' and the graphic features H3 / H4 / H5 are input into the Refining Decoder for layer-by-layer decoding, and the object features (query embedding) of the implicit tracking results are output.
[0163] (6) The object features output by RefiningDecoder are input into Predict Head for linear processing to obtain the final detection result.
[0164] From the above, it can be seen that after obtaining the video to be detected and the detection text corresponding to the video to be detected, the embodiment of the present application performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, and then calculates the feature correlation between the visual features and the text features at each pixel position, and based on the feature correlation, fuses the visual features with the text features to obtain visual text features, and then detects at least one target object in the video frame to be detected based on the visual text features and the visual features; because the scheme can calculate the feature correlation between the visual features and the text features at each pixel position after extracting the video features and the text features, and fuse the visual features with the text features through the feature correlation, and adopts a pixel-level image-text feature fusion method in the fusion process, so that fine-grained fusion between visual features and text features can be achieved, and therefore, the accuracy of object detection can be improved.
[0165] According to the method described in the above embodiment, the following is further described in detail with examples.
[0166] In this embodiment, the object detection device is specifically integrated in an electronic device, and the electronic device is a server.
[0167] like Figure 7 As shown, an object detection method, the specific process is as follows:
[0168] 201. The server obtains a video to be detected and a detection text corresponding to the video to be detected.
[0169] For example, the server can receive a video to be detected and a detection text corresponding to the video to be detected uploaded by a terminal or a client, or at least one original video can be obtained from a video platform or a network as the video to be detected, and the detection text corresponding to the video to be detected can be screened out from a text collection, or, based on a target detection task, at least one detection text can be screened out from a text collection to obtain a text collection, and at least one video to be detected corresponding to the target detection task can be screened out from a content platform, and the detection text corresponding to each video to be detected can be determined in the text collection, or, when the number of videos to be detected and the detection text exceeds a preset number threshold, an object detection request can also be received, and the object detection request carries the storage address of the video to be detected and the detection text corresponding to the video to be detected, and based on the storage address, the video to be detected and the detection text corresponding to the video to be detected are obtained, and so on.
[0170] 202. The server extracts features from the video frame to be detected to obtain visual features of at least one dimension.
[0171] For example, the server may use the RN50 version image encoder to perform multi-scale feature extraction on the video frame to be detected, and use the visual features output by different network layers as the initial visual features of at least one feature scale.
[0172] The server can filter out the initial visual features with the deepest feature scale from the initial visual features to obtain the target visual features, or can sort the initial visual features based on the feature scale, and based on the sorting results, filter out the initial visual features with the deepest feature scale from the initial visual features to obtain the target visual features, and so on.
[0173] The server may use a flatten layer in a feature extraction network to flatten the target initial visual feature, thereby obtaining an adjusted visual feature. The adjusted visual feature is concatenated with the sine and cosine position encoding, thereby obtaining a concatenated visual feature, wherein the concatenated visual feature includes at least one visual sub-feature. The visual sub-feature is self-attention weighted, thereby obtaining a weighted visual feature. The target initial visual feature in the initial visual feature is replaced with the weighted visual feature, thereby obtaining a candidate visual feature of at least one dimension.
[0174] The server may sort the candidate visual features based on their feature sizes to obtain a first size sorting result, select a candidate visual feature with the smallest feature size from the candidate visual features to obtain a current candidate visual feature, and use the current candidate visual feature as a first candidate interactive feature, and select the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result to obtain a first feature to be interacted with the current candidate visual feature.
[0175] The server can use 1*1 convolution to convert the number of feature channels of the current candidate visual feature into a preset number of channels to obtain the converted visual feature. The target feature size of the first feature to be interacted with is obtained, and based on the target feature size, the feature size of the converted visual feature is interpolated to the same as that of the first feature to be interacted with by bilinear interpolation, thereby obtaining the enlarged visual feature. The number of feature channels of the first feature to be interacted with is converted into a preset number of channels by 1*1 convolution, thereby obtaining the converted first feature to be interacted with. The enlarged visual feature is spliced with the converted first feature to be interacted with in the dimension of the feature channel, thereby obtaining the second candidate interactive feature.
[0176] The server may use the first feature to be interacted as the current candidate visual feature, and then return to perform the step of selecting the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result, until the candidate visual features complete feature interaction, thereby obtaining at least one candidate interactive feature. The candidate interactive feature may include the first candidate interactive feature and the second candidate interactive feature.
[0177] The server may sort the candidate interaction features from large to small according to feature size to obtain a second size sorting result. The largest candidate interaction feature is selected from the candidate interaction features to obtain the current interaction feature, and features of different feature channels in the current interaction feature are fused to obtain the first interaction feature. Based on the second size sorting result, the next candidate interaction feature of the current interaction feature is selected from the candidate interaction features to obtain the second to-be-interacted feature of the current interaction feature.
[0178] The server may perform feature channel conversion on the current interactive feature to obtain a converted interactive feature with a preset number of channels, and halve the feature size of the converted visual feature to obtain an adjusted interactive feature. Perform feature channel conversion on the second feature to be interacted with to obtain a converted second feature to be interacted with with a preset number of channels, and concatenate the adjusted interactive feature with the converted second feature to be interacted with to obtain a second interactive feature.
[0179] The server uses the second feature to be interacted as the current interaction feature, returns to execute the step of selecting the next candidate interaction feature of the current interaction feature from the candidate interaction features based on the sorting result of the second dimension, until all candidate interaction features complete feature interaction, and obtains at least one interaction feature, which includes the first interaction feature and the second interaction feature. Each interaction feature is regarded as a visual feature of one dimension.
[0180] 203. The server extracts features from the detected text to obtain text features.
[0181] For example, the server may use the text feature extractor of CLIP to extract text embedding from the detected text, thereby obtaining text features.
[0182] 204. The server maps the visual features and the text features into the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features.
[0183] For example, the server can identify the number of feature channels in the visual features to obtain the number of visual channels, and identify the number of feature channels in the text features to obtain the number of text channels. Based on the number of visual channels and the number of text channels, the server can determine the target number of channels in the feature space corresponding to each dimension, fuse the text word features to obtain fused text features, adjust the number of channels of the visual features and the number of channels of the fused text features to the target number of channels, respectively, and obtain the target visual features of each dimension and the target text features corresponding to the target visual features.
[0184] 205. The server extracts the visual sub-features corresponding to each pixel position from the target visual features, and calculates the feature correlation between the visual sub-features and the target text features.
[0185] For example, the server can obtain the pixel position of each pixel in the video frame to be detected, extract the visual sub-features corresponding to each pixel position from the target visual features, extract the features under each feature channel from the visual sub-features, obtain the visual channel features, and filter out the features under the feature channels corresponding to the visual channel features from the target text features to obtain the text channel features.
[0186] The server fuses the visual channel features with the text channel features to obtain the fused channel features corresponding to each feature channel, accumulates the fused channel features at each pixel position to obtain the target fused features corresponding to each pixel position, fuses the visual sub-features with the target text features to obtain the reference fused features, and calculates the ratio between the reference fused features and the target fused features to obtain the feature correlation corresponding to each pixel position, which can be specifically shown in formula (1).
[0187] 206. The server fuses the visual feature with the text feature based on the feature relevance to obtain the visual text feature.
[0188] For example, the server selects the target visual sub-features corresponding to the feature relevance from the visual sub-features, and weights the target visual sub-features based on the feature relevance, thereby obtaining the initial visual text features corresponding to each pixel position, which can be shown in formula (2). The initial visual text features of the same dimension are fused to obtain the visual text features of each dimension.
[0189] 207. The server detects at least one target object in the video frame to be detected according to the visual text features and the visual features.
[0190] For example, the server may sort the visual features based on the dimensions of the visual features to obtain a sorting result.
[0191] Based on the sorting results, the server can select the visual feature with the deepest feature scale from the visual features as the current visual feature corresponding to the target dimension, or, based on the sorting results, select the visual feature with the smallest feature size from the visual features as the current visual feature corresponding to the target dimension, and so on.
[0192] The server can perform self-attention weighting on the object sub-features to obtain candidate object features, perform cross-attention weighting on the candidate object features, the current visual text features, and the current visual features to obtain the current object features, and map the current object features to the preset object feature space to obtain the initial object features. The updated initial object features are used as the preset object features, and then the step of updating the preset object features based on the current visual text features and the current visual features is returned until the preset number of updates is reached to obtain the updated object features.
[0193] The server may use the updated object features as preset object features. Then, the server returns to the step of selecting the visual features of the target dimension from the visual features based on the sorting result until each visual feature is the current visual feature, thereby obtaining the object features.
[0194] The server may use an object prediction network (predict head) to perform linear processing on the object features, thereby obtaining the confidence and object attribute information of at least one candidate object. At least one candidate object whose object confidence exceeds a preset confidence threshold is selected from the candidate objects as the target object. Object attribute information corresponding to the target object is selected from the object attribute information to obtain the target object attribute information.
[0195] Optionally, in some embodiments, the server can filter out target video frames from the video frames to be detected in the video to be detected based on the target object attribute information, identify at least one target area in the target video frame, and edit the content of the target area; or, based on the target object attribute information, identify at least one target area in the video frames to be detected in the video to be detected, rewrite the target area, and achieve consistent rewriting of content across frames based on the timing of the target area, and so on.
[0196] From the above, it can be seen that after obtaining the video to be detected and the detection text corresponding to the video to be detected, the server of this embodiment performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, and then calculates the feature correlation between the visual features and the text features at each pixel position, and based on the feature correlation, fuses the visual features with the text features to obtain visual text features, and then detects at least one target object in the video frame to be detected based on the visual text features and the visual features; because this scheme can calculate the feature correlation between the visual features and the text features at each pixel position after extracting the video features and the text features, and fuse the visual features with the text features through the feature correlation, and adopts a pixel-level image-text feature fusion method in the fusion process, so that fine-grained fusion between visual features and text features can be achieved, and therefore, the accuracy of object detection can be improved.
[0197] In order to better implement the above method, an embodiment of the present application also provides an object detection device, which can be integrated into an electronic device, such as a server or a terminal. The terminal may include a tablet computer, a laptop computer and / or a personal computer.
[0198] For example, Figure 8 As shown, the object detection device may include an acquisition unit 301, an extraction unit 302, a calculation unit 303, a fusion unit 304 and a detection unit 305, as follows:
[0199] (1) Acquisition unit 301;
[0200] The acquisition unit 301 is used to acquire a video to be detected and a detection text corresponding to the video to be detected, where the video to be detected includes at least one video frame to be detected.
[0201] For example, the acquisition unit 301 may be specifically configured to acquire at least one video as a video to be detected, and acquire a detection text corresponding to the video to be detected.
[0202] (2) Extraction unit 302;
[0203] The extraction unit 302 is used to perform feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and to perform feature extraction on the detected text to obtain text features.
[0204] For example, the extraction unit 302 can be specifically used to perform multi-scale feature extraction on the video frame to be detected, obtain initial visual features of at least one feature scale, preprocess the initial visual features to obtain candidate visual features of at least one dimension, perform feature interaction on the candidate visual features according to a preset interaction direction to obtain visual features of at least one dimension, and perform feature extraction on the detected text to obtain text features.
[0205] (3) a computing unit 303;
[0206] The calculation unit 303 is used to calculate the feature correlation between the visual feature and the text feature at each pixel position, where the feature correlation indicates the correlation degree between the visual feature and the text feature at the same pixel position.
[0207] For example, the computing unit 303 can be specifically used to map visual features and text features into the same feature space, obtain the target visual features of each dimension and the target text features corresponding to the target visual features, extract the visual sub-features corresponding to each pixel position in the target visual features, and calculate the feature correlation between the visual sub-features and the target text features.
[0208] (4) fusion unit 304;
[0209] The fusion unit 304 is used to fuse the visual feature with the text feature based on the feature correlation to obtain the visual text feature.
[0210] For example, the fusion unit 304 may be specifically configured to weight the visual sub-features based on feature relevance to obtain an initial visual text feature corresponding to each pixel position, and fuse the initial visual text features of the same dimension to obtain a visual text feature of each dimension.
[0211] (5) Detection unit 305;
[0212] The detection unit 305 is used to detect at least one target object in the video frame to be detected according to the visual text feature and the visual feature.
[0213] For example, the detection unit 305 can be specifically used to sort the visual features based on the dimensions of the visual features, and filter out the visual features of the target dimension from the visual features based on the sorting results to obtain the current visual features, filter out the visual text features corresponding to the target dimension from the visual text features to obtain the current visual text features, extract object features in the video frame to be detected based on the current visual text features and the current visual features, and determine at least one target object in the video frame to be detected based on the object features.
[0214] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0215] From the above, it can be seen that in the embodiment of the present application, after the acquisition unit 301 acquires the video to be detected and the detection text corresponding to the video to be detected, the extraction unit 302 performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, and then the calculation unit 303 calculates the feature correlation between the visual features and the text features at each pixel position, and the fusion unit 304 fuses the visual features with the text features based on the feature correlation to obtain visual text features, and then the detection unit 305 detects at least one target object in the video frame to be detected based on the visual text features and the visual features; because the scheme can calculate the feature correlation between the visual features and the text features at each pixel position after extracting the video features and the text features, and fuse the visual features with the text features through the feature correlation, and adopts the pixel-level image-text feature fusion method in the fusion process, so that fine-grained fusion between visual features and text features can be achieved, and therefore, the accuracy of object detection can be improved.
[0216] The present application also provides an electronic device, such as Fig. 9 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0217] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0218] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.
[0219] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0220] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.
[0221] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0222] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:
[0223] A video to be detected and a detection text corresponding to the video to be detected are obtained, wherein the video to be detected includes at least one frame of the video to be detected, feature extraction is performed on the video frame to be detected to obtain visual features of at least one dimension, and feature extraction is performed on the detection text to obtain text features, and feature correlations between the visual features and the text features at each pixel position are calculated, wherein the feature correlation indicates the degree of correlation between the visual features and the text features at the same pixel position, and based on the feature correlation, the visual features are fused with the text features to obtain visual text features, and at least one target object is detected in the video frame to be detected according to the visual text features and the visual features.
[0224] For example, the electronic device can obtain at least one video as a video to be detected, and obtain the detection text corresponding to the video to be detected. Multi-scale feature extraction is performed on the video frame to be detected to obtain the initial visual feature of at least one feature scale, and the initial visual feature is preprocessed to obtain candidate visual features of at least one dimension. According to the preset interaction direction, the candidate visual features are subjected to feature interaction to obtain visual features of at least one dimension, and feature extraction is performed on the detection text to obtain text features. The visual features and text features are mapped to the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features. The visual sub-features corresponding to each pixel position are extracted from the target visual features, and the feature correlation between the visual sub-features and the target text features is calculated. Based on the feature correlation, the visual sub-features are weighted to obtain the initial visual text features corresponding to each pixel position, and the initial visual text features of the same dimension are fused to obtain the visual text features of each dimension. Based on the dimension of the visual features, the visual features are sorted, and based on the sorting result, the visual features of the target dimension are screened out from the visual features to obtain the current visual features, and the visual text features corresponding to the target dimension are screened out from the visual text features to obtain the current visual text features, and object features are extracted from the video frame to be detected based on the current visual text features and the current visual features, and based on the object features, at least one target object in the video frame to be detected is determined.
[0225] The specific implementation of the above operations can be found in the previous embodiments and will not be described in detail here.
[0226] From the above, it can be seen that after obtaining the video to be detected and the detection text corresponding to the video to be detected, the embodiment of the present application performs feature extraction on the video frame to be detected in the video to be detected to obtain visual features of at least one dimension, and performs feature extraction on the detection text to obtain text features, and then calculates the feature correlation between the visual features and the text features at each pixel position, and based on the feature correlation, fuses the visual features with the text features to obtain visual text features, and then detects at least one target object in the video frame to be detected based on the visual text features and the visual features; because the scheme can calculate the feature correlation between the visual features and the text features at each pixel position after extracting the video features and the text features, and fuse the visual features with the text features through the feature correlation, and adopts a pixel-level image-text feature fusion method in the fusion process, so that fine-grained fusion between visual features and text features can be achieved, and therefore, the accuracy of object detection can be improved.
[0227] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0228] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any object detection method provided in the embodiment of the present application. For example, the instructions can execute the following steps:
[0229] A video to be detected and a detection text corresponding to the video to be detected are obtained, wherein the video to be detected includes at least one frame of the video to be detected, feature extraction is performed on the video frame to be detected to obtain visual features of at least one dimension, and feature extraction is performed on the detection text to obtain text features, and feature correlations between the visual features and the text features at each pixel position are calculated, wherein the feature correlation indicates the degree of correlation between the visual features and the text features at the same pixel position, and based on the feature correlation, the visual features are fused with the text features to obtain visual text features, and at least one target object is detected in the video frame to be detected according to the visual text features and the visual features.
[0230] For example, at least one video is obtained as the video to be detected, and the detection text corresponding to the video to be detected is obtained. Multi-scale feature extraction is performed on the video frame to be detected to obtain the initial visual feature of at least one feature scale, and the initial visual feature is preprocessed to obtain candidate visual features of at least one dimension. According to the preset interaction direction, the candidate visual features are subjected to feature interaction to obtain the visual features of at least one dimension, and the detection text is subjected to feature extraction to obtain text features. The visual features and the text features are mapped to the same feature space to obtain the target visual features of each dimension and the target text features corresponding to the target visual features. The visual sub-features corresponding to each pixel position are extracted from the target visual features, and the feature correlation between the visual sub-features and the target text features is calculated. Based on the feature correlation, the visual sub-features are weighted to obtain the initial visual text features corresponding to each pixel position, and the initial visual text features of the same dimension are fused to obtain the visual text features of each dimension. Based on the dimension of the visual features, the visual features are sorted, and based on the sorting result, the visual features of the target dimension are screened out from the visual features to obtain the current visual features, and the visual text features corresponding to the target dimension are screened out from the visual text features to obtain the current visual text features, and object features are extracted from the video frame to be detected based on the current visual text features and the current visual features, and based on the object features, at least one target object in the video frame to be detected is determined.
[0231] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0232] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0233] Since the instructions stored in the computer-readable storage medium can execute the steps in any object detection method provided in the embodiments of the present application, the beneficial effects that can be achieved by any object detection method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0234] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the methods provided in various optional implementations of the above-mentioned object detection or target tracking aspects.
[0235] The above is a detailed introduction to an object detection method and related equipment provided in an embodiment of the present application, which may include an object detection device, an electronic device, a computer program product and a computer-readable storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. An object detection method, characterized in that: include: Acquire a video to be detected and a detection text corresponding to the video to be detected, wherein the video to be detected includes at least one video frame to be detected; Performing feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and performing feature extraction on the detected text to obtain text features; Calculating a feature correlation between the visual feature and the text feature at each pixel position, wherein the feature correlation indicates a correlation degree between the visual feature and the text feature at the same pixel position; Based on the feature correlation, the visual feature and the text feature are fused to obtain a visual text feature; At least one target object is detected in the to-be-detected video frame according to the visual text feature and the visual feature.
2. The object detection method according to claim 1, characterized in that: The calculating the feature correlation between the visual feature and the text feature at each pixel position includes: Mapping the visual features and the text features into the same feature space to obtain target visual features of each dimension and target text features corresponding to the target visual features; A visual sub-feature corresponding to each pixel position is extracted from the target visual feature, and a feature correlation between the visual sub-feature and the target text feature is calculated.
3. The object detection method according to claim 2, characterized in that: The text feature includes at least one text word feature, and the visual feature and the text feature are mapped to the same feature space to obtain a target visual feature of each dimension and a target text feature corresponding to the target visual feature, including: Identifying the number of feature channels in the visual features to obtain the number of visual channels, and identifying the number of feature channels in the text features to obtain the number of text channels; Based on the number of visual channels and the number of text channels, determining the target number of channels in the feature space corresponding to each dimension, and fusing the text word features to obtain fused text features; The number of channels of the visual feature and the number of channels of the fused text feature are respectively adjusted to the target number of channels, so as to obtain the target visual feature of each dimension and the target text feature corresponding to the target visual feature.
4. The object detection method according to claim 2, characterized in that: The calculating the feature correlation between the visual sub-feature and the target text feature includes: Extracting features under each feature channel from the visual sub-features to obtain visual channel features; Screening out features under the feature channel corresponding to the visual channel feature from the target text features to obtain text channel features; Based on the visual channel feature and the text channel feature, a feature correlation between the visual sub-feature and the target text feature is determined.
5. The object detection method according to claim 4, characterized in that: The determining, based on the visual channel feature and the text feature channel, the feature correlation between the visual sub-feature and the target text feature comprises: The visual channel features are fused with the text channel features to obtain fused channel features corresponding to each feature channel; Accumulating the fusion channel features at each pixel position to obtain the target fusion feature corresponding to each pixel position; The visual sub-feature is fused with the target text feature to obtain a reference fused feature, and the ratio between the reference fused feature and the target fused feature is calculated to obtain the feature correlation corresponding to each pixel position.
6. The object detection method according to claim 2, characterized in that: The step of fusing the visual feature with the text feature based on the feature correlation to obtain a visual text feature includes: Based on the feature relevance, weighting the visual sub-features to obtain an initial visual text feature corresponding to each pixel position; The initial visual text features of the same dimension are fused to obtain visual text features of each dimension.
7. The object detection method according to any one of claims 1 to 6, characterized in that: The detecting at least one target object in the to-be-detected video frame according to the visual text feature and the visual feature comprises: Based on the dimension of the visual feature, the visual features are sorted, and based on the sorting result, the visual features of the target dimension are screened out from the visual features to obtain the current visual feature; Selecting the visual text features corresponding to the target dimension from the visual text features to obtain the current visual text features; According to the current visual text feature and the current visual feature, an object feature is extracted from the video frame to be detected, and based on the object feature, at least one target object in the video frame to be detected is determined.
8. The object detection method according to claim 7, characterized in that: The extracting object features from the to-be-detected video frame according to the current visual text features and the current visual features comprises: Based on the current visual text feature and the current visual feature, the preset object feature is updated, and the updated object feature is used as the preset object feature; Return to the step of filtering out the visual features of the target dimension from the visual features based on the sorting result, until each visual feature is the current visual feature, to obtain the object feature.
9. The object detection method according to claim 8, characterized in that: The updating of the preset object feature based on the current visual text feature and the current visual feature includes: Based on the current visual text feature and the current visual feature, the preset object feature is updated to obtain an initial object feature, and the initial object feature is used as the preset object feature; Return to the step of updating the preset object feature based on the current visual text feature and the current visual feature until a preset number of updates is reached to obtain an updated object feature.
10. The object detection method according to claim 9, characterized in that: The preset object feature includes at least one object sub-feature, and the updating of the preset object feature based on the current visual text feature and the current visual feature to obtain the initial object feature includes: Performing self-attention weighting on the object sub-features to obtain candidate object features; Performing cross-attention weighting on the candidate object features, the current visual text features and the current visual features to obtain the current object features; The current object feature is mapped to a preset object feature space to obtain an initial object feature.
11. The object detection method according to any one of claims 1 to 6, characterized in that: The extracting features of the video frame to be detected to obtain visual features of at least one dimension includes: Performing multi-scale feature extraction on the video frame to be detected to obtain an initial visual feature of at least one feature scale; Preprocessing the initial visual features to obtain candidate visual features of at least one dimension; According to a preset interaction direction, feature interaction is performed on the candidate visual features to obtain visual features of at least one dimension.
12. The object detection method according to claim 11, characterized in that: The preprocessing of the initial visual features to obtain candidate visual features of at least one dimension includes: Based on the feature scale, a target initial visual feature is selected from the initial visual features, and a size of the target initial visual feature is adjusted to obtain an adjusted visual feature; Splicing the adjusted visual feature with the preset position feature to obtain a spliced visual feature, wherein the spliced visual feature includes at least one visual sub-feature; The visual sub-features are self-attention weighted to obtain weighted visual features, and the initial visual features are updated based on the weighted visual features to obtain candidate visual features of at least one dimension.
13. The object detection method according to claim 11, characterized in that: The preset interaction direction includes a first interaction direction and a second interaction direction, and the feature interaction is performed on the candidate visual features according to the preset interaction direction to obtain the visual features of at least one dimension, including: sorting the candidate visual features based on the feature sizes of the candidate visual features to obtain a first size sorting result; According to the first size sorting result, performing feature interaction on the candidate visual features according to the first interaction direction to obtain at least one candidate interaction feature; sorting the candidate interaction features based on the feature sizes of the candidate interaction features to obtain a second size sorting result; According to the second dimension sorting result, feature interaction is performed on the candidate interaction features according to the second interaction direction to obtain at least one interaction feature, and each interaction feature is used as a visual feature of one dimension.
14. The object detection method according to claim 13, characterized in that: The step of performing feature interaction on the candidate visual features according to the first interaction direction according to the first size sorting result to obtain at least one candidate interaction feature includes: Selecting a candidate visual feature with the smallest feature size from the candidate visual features to obtain a current candidate visual feature, and using the current candidate visual feature as a first candidate interaction feature; Based on the first size sorting result, selecting a next candidate visual feature of the current candidate visual feature from the candidate visual features to obtain a first feature to be interacted with of the current candidate visual feature; The current candidate visual feature is merged with the first feature to be interacted with to obtain a second candidate interactive feature, and the first feature to be interacted with is used as the current candidate visual feature; Return to execute the step of screening out the next candidate visual feature of the current candidate visual feature from the candidate visual features based on the first size sorting result, until all candidate visual features complete feature interaction, and obtain at least one candidate interaction feature, wherein the candidate interaction features include the first candidate interaction feature and the second candidate interaction feature.
15. The object detection method according to claim 14, characterized in that: The step of fusing the current candidate visual feature with the first feature to be interacted with to obtain a second candidate interaction feature includes: Performing feature channel conversion on the current candidate visual feature to obtain converted visual features with a preset number of channels; Amplifying the feature size of the converted visual feature to obtain an amplified visual feature, wherein the feature size of the amplified visual feature is the same as the feature size of the first feature to be interacted with; Performing feature channel conversion on the first feature to be interacted with to obtain the converted first features to be interacted with after the preset number of channels; The enlarged visual feature is concatenated with the converted first feature to be interacted with to obtain a second candidate interaction feature.
16. The object detection method according to claim 13, characterized in that: The step of performing feature interaction on the candidate interaction features according to the second interaction direction according to the second size sorting result to obtain at least one interaction feature includes: Selecting a candidate interaction feature with the largest feature size from the candidate interaction features to obtain a current interaction feature, and fusing features of different feature channels in the current interaction feature to obtain a first interaction feature; Based on the second size sorting result, screening out the next candidate interaction feature of the current interaction feature from the candidate interaction features, and obtaining a second to-be-interacted feature of the current interaction feature; The current interaction feature is merged with the second feature to be interacted with to obtain a second interaction feature, and the second feature to be interacted with is used as the current interaction feature; Return to executing the step of sorting the result based on the second size and selecting the next candidate interaction feature of the current interaction feature from the candidate interaction features until all candidate interaction features complete feature interaction, and obtain at least one interaction feature, wherein the interaction features include the first interaction feature and the second interaction feature.
17. An object detection device, characterized in that: include: An acquisition unit, used for acquiring a video to be detected and a detection text corresponding to the video to be detected, wherein the video to be detected includes at least one video frame to be detected; An extraction unit, used to perform feature extraction on the video frame to be detected to obtain visual features of at least one dimension, and perform feature extraction on the detected text to obtain text features; A calculation unit, used for calculating the feature correlation between the visual feature and the text feature at each pixel position, wherein the feature correlation indicates the correlation degree between the visual feature and the text feature at the same pixel position; A fusion unit, configured to fuse the visual feature with the text feature based on the feature correlation to obtain a visual text feature; A detection unit is used to detect at least one target object in the video frame to be detected according to the visual text feature and the visual feature.
18. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the object detection method according to any one of claims 1 to 16.
19. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the object detection method according to any one of claims 1 to 16 are implemented.
20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the object detection method according to any one of claims 1 to 16.
Citation Information
Patent Citations
Remote sensing image visual positioning method based on text guidance
CN116958829A
Video object positioning method and device, storage medium and program product
CN117351382A
Video target tracking method based on language description
CN118115526A
Target positioning method and device guided by natural language and storage medium
CN118212297A
Natural language object tracking
US20180129742A1
