Video processing method, electronic device, storage medium and computer program product

Abstract

The present application provides a video processing method, electronic device, storage medium, and computer program product. The method includes: obtaining target image features corresponding to each of multiple frame sequences of a video to be processed; performing target detection based on the target image features corresponding to the current frame sequence; position encoding at least part of the position information in the initial target detection result to obtain a first position encoding feature; obtaining an image embedding feature corresponding to at least part of the position information in the initial target detection result; fusing the first position encoding feature with the image embedding feature to obtain a current query feature; generating a target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature; decoding based on the target image feature and the target query feature to obtain an updated query feature corresponding to the current frame sequence; and determining a final target detection result based on the updated query feature. This method can achieve modeling of video temporal relationships.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and more specifically to a video processing method, electronic equipment, storage medium, and computer program product. Background Art

[0002] Currently, video object detection and video instance segmentation are commonly used video processing technologies in the field of video processing. Below, we use video instance segmentation as an example to describe the problems existing in these video processing technologies. It should be noted that video object detection faces similar problems, except that video object detection primarily detects the target object in the video, while video instance segmentation goes further by segmenting the target object (i.e., instance) in the video. Video instance segmentation tasks often require detecting, tracking, and segmenting all instances in the video. Therefore, video instance segmentation requires not only the model's ability to recognize and segment instances in a single frame, but also the ability to connect and continuously track instances in the video.

[0003] The existing video instance segmentation methods mainly include the following schemes: using a feature extraction network for single-frame images to extract features of each video frame in the video, then performing instance detection and segmentation on each video frame separately, and after obtaining the results of a single frame, an instance matching algorithm between adjacent frames is used to obtain the instance sequence of the entire video. The feature extraction network for single-frame images is suitable for feature extraction of single-frame images, but lacks modeling of the temporal relationship between frames, making it difficult to achieve effective detection and recognition of video instance sequences (video target detection has similar problems). Therefore, a new video processing solution is needed to solve the above technical problems. Summary of the Invention

[0004] In view of the above problems, the present application is proposed. The present application provides a video processing method, an electronic device, a storage medium and a computer program product.

[0005] According to one aspect of the present application, a video processing method is provided, comprising: obtaining target image features corresponding to each of a plurality of frame sequences of a video to be processed, each frame sequence in the plurality of frame sequences including one or more video frames, and the target image features corresponding to any frame sequence including the target image features corresponding to one or more video frames in the corresponding frame sequence; for any current frame sequence in the video to be processed, performing the following frame sequence processing operations: performing target detection based on the target image features corresponding to the current frame sequence to obtain an initial target detection result corresponding to the current frame sequence; position encoding at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature; obtaining an image embedding feature corresponding to at least part of the position information in the initial target detection result corresponding to the current frame sequence; encoding the first position encoding feature; The features are fused with the image embedding features to obtain the current query features corresponding to the current frame sequence; based on at least part of the feature vectors in the updated query features corresponding to the previous frame sequence and the current query features, a target query feature is generated, wherein the current query feature, the updated query feature and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object; based on the target image features corresponding to the current frame sequence and the target query features, decoding is performed to obtain the updated query features corresponding to the current frame sequence; based on the updated query features corresponding to the current frame sequence, a final target detection result corresponding to the current frame sequence is determined; wherein the initial target detection result includes the initial position information of the target object in each video frame in the corresponding frame sequence, and the final target detection result includes the final position information of the target object in each video frame in the corresponding frame sequence.

[0006] Exemplarily, the initial position information is used to indicate the predicted position of the initial detection frame where the target object is located, and the final position information is used to indicate the predicted position of the final detection frame where the target object is located. The initial target detection result also includes a confidence level corresponding to each initial detection frame, and the final target detection result also includes a confidence level corresponding to each final detection frame. Before position encoding is performed on at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain the first position encoding feature, the frame sequence processing operation further includes: selecting the initial position corresponding to the initial detection frame whose confidence level in the initial target detection result corresponding to the current frame sequence is greater than or equal to the first confidence threshold. Information is used as at least part of the information in the initial target detection result corresponding to the current frame sequence; and / or, before generating the target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, the frame sequence processing operation also includes: selecting a final detection frame with a confidence level less than a second confidence threshold in the final target detection result corresponding to the previous frame sequence, and using the feature vectors in the updated query feature corresponding to the previous frame sequence except for the specific feature vector as at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence, wherein the specific feature vector is the feature vector corresponding to the selected final detection frame.

[0007] Exemplarily, in the case where each frame sequence includes multiple video frames, in any two adjacent frame sequences, the video frames included in the first frame sequence are partially identical to the video frames included in the second frame sequence.

[0008] Exemplarily, obtaining target image features corresponding to multiple frame sequences of a video to be processed includes: for any current frame sequence in the video to be processed, performing feature extraction on each video frame in the current frame sequence to obtain initial image features corresponding to the current frame sequence, the initial image features corresponding to the current frame sequence include initial image features corresponding to one or more video frames in the current frame sequence; fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain memory token features corresponding to the current frame sequence; fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the current frame sequence to obtain target image features corresponding to the current frame sequence.

[0009] Exemplarily, the initial image features corresponding to the current frame sequence are fused with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain the memory token features corresponding to the current frame sequence, including: position encoding the initial image features corresponding to the current frame sequence to obtain a second position encoding feature, and the second position encoding feature has the same dimension as the initial image features corresponding to the current frame sequence; merging the second position encoding feature with the initial image features corresponding to the current frame sequence to obtain a merged feature; performing an attention mechanism operation on the merged feature and the memory token features corresponding to the previous frame sequence to obtain the memory token features corresponding to the current frame sequence; fusing the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence, including: performing an attention mechanism operation on the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence.

[0010] Exemplarily, the final position information is used to indicate the predicted position of the final detection frame where the target object is located. After determining the final target detection result of the current frame sequence based on the updated query feature corresponding to the current frame sequence, the frame sequence processing operation also includes: based on the final target detection result of the current frame sequence, mapping at least part of the final detection frame to the target image feature corresponding to the current frame sequence, and obtaining local image features corresponding to at least part of the final detection frames; using the local image feature corresponding to any final detection frame as a convolution kernel, convolving the target image feature corresponding to the current frame sequence, and obtaining mask information corresponding to the final detection frame, the mask information is used to indicate the position of the mask of the target object contained in the corresponding final detection frame.

[0011] Exemplarily, target detection is performed based on the target image features corresponding to the current frame sequence, and the initial target detection result corresponding to the current frame sequence is obtained by the target detection module in the video processing model. Based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, the target query feature is generated by the decoding module in the video processing model. The video processing model is trained in the following manner: labeled target detection results and target image features corresponding to multiple frame sequences of the sample video are obtained, and the labeled target detection results include the labeled position information of the target object in each video frame in the corresponding frame sequence; for any current frame sequence in the sample video, the video processing model is used to perform frame sequence processing operations to obtain the predicted target detection result corresponding to the current frame sequence; based on the predicted target detection results and labeled target detection results corresponding to each of the multiple frame sequences in the sample video, the prediction loss is calculated; and the parameters in the video processing model are optimized based on the prediction loss.

[0012] According to another aspect of the present application, an electronic device is provided, including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned video processing method when the processor is running.

[0013] According to another aspect of the present application, a storage medium is provided, on which program instructions are stored, wherein the program instructions are used to execute the above-mentioned video processing method when running.

[0014] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program, wherein the computer program is used to execute the above-mentioned video processing method when running.

[0015] According to the video processing method, electronic device, storage medium and computer program product of the embodiment of the present application, position encoding is performed based on at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature, and then the current query feature corresponding to the current frame sequence is obtained based on the first position encoding feature. The method further generates a target query feature based on at least part of the feature vector in the updated query feature corresponding to the previous frame sequence and the current query feature, and performs a query based on the generated target query feature to obtain the final target detection result of the current frame sequence. The method can integrate the target detection information of the previous frame sequence as a priori into the query feature of the current frame sequence, so that the temporal relationship of the video can be modeled, which helps to achieve continuous search and tracking of target objects in the video frame, and thus can effectively and accurately detect and identify the video instance sequence. In addition, the method can also perform target detection through the target image features corresponding to each of the multiple frame sequences, thereby ensuring a relatively accurate single-frame target object detection (and instance segmentation) effect. Therefore, the above scheme allows for relatively accurate single-frame target detection (and instance segmentation) of video frames, while also making the final target detection results obtained continuous and consistent in time sequence, which helps to obtain relatively accurate location information of target objects in both time and space dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1A schematic block diagram showing an example electronic device for implementing the video processing method and apparatus according to an embodiment of the present application;

[0018] Figure 2 A schematic flowchart showing a video processing method according to an embodiment of the present application is shown;

[0019] Figure 3 A schematic diagram illustrating a frame sequence processing operation according to an embodiment of the present application;

[0020] Figure 4 A schematic block diagram showing a video processing device according to an embodiment of the present application; and

[0021] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, image processing, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0023] In order to make the purpose, technical solutions and advantages of the present application more apparent, the following is a detailed description of example embodiments of the present application with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0024] The embodiments of the present application provide a video processing method, electronic device, storage medium, and computer program product. According to the video processing method of the embodiments of the present application, target detection information of a previous frame sequence can be incorporated into the query features of the current frame sequence as a priori, thereby modeling the temporal relationship of the video and facilitating the acquisition of relatively accurate position information of the target object in both the temporal and spatial dimensions. The video processing technology according to the embodiments of the present application can be applied to any field involving video target detection.

[0025] First, refer to Figure 1 An example electronic device 100 for implementing the video processing method and apparatus according to an embodiment of the present application is described.

[0026] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may further include an input device 106, an output device 108, and an image acquisition device 110, and these components are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as needed.

[0027] The processor 102 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic array (PLA), or a microprocessor. The processor 102 can be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or one or more combinations of other forms of processing units with data processing capabilities and / or instruction execution capabilities, and can control other components in the electronic device 100 to perform desired functions.

[0028] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions and / or other desired functions in the embodiments of the present application (implemented by the processor) described below. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.

[0029] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0030] The output device 108 can output various information (e.g., images and / or sounds) to the outside (e.g., a user) and can include one or more of a display, a speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (e.g., a touch screen).

[0031] The image acquisition device 110 can capture images and store them in the storage device 104 for use by other components. The image acquisition device 110 can be a separate camera or a camera in a mobile terminal. It should be understood that the image acquisition device 110 is merely an example, and the electronic device 100 may not include the image acquisition device 110. In this case, other devices with image acquisition capabilities can be used to capture images and transmit the captured images to the electronic device 100.

[0032] For example, an example electronic device for implementing the video processing method and apparatus according to the embodiments of the present application can be implemented on a device such as a personal computer, a terminal device, an attendance machine, a panel machine, a camera, or a remote server. The terminal device includes, but is not limited to, a tablet computer, a mobile phone, a PDA (Personal Digital Assistant), a touch-screen all-in-one machine, a wearable device, and the like.

[0033] Below, we will refer to Figure 2 A video processing method according to an embodiment of the present application is described. Figure 2 FIG. 2 is a schematic flow chart of a video processing method 200 according to an embodiment of the present application. Figure 2As shown, the video processing method 200 includes the following steps S210 and S220.

[0034] Step S210, obtaining target image features corresponding to each of multiple frame sequences of the video to be processed, each frame sequence in the multiple frame sequences includes one or more video frames, and the target image features corresponding to any frame sequence include target image features corresponding to one or more video frames in the corresponding frame sequence.

[0035] The video to be processed can come from an external device and be transmitted by the external device to the electronic device 100 for video processing. Alternatively, the video to be processed can be acquired by the electronic device 100 itself. For example, the electronic device 100 can utilize an image acquisition device 110 (e.g., an independent camera) to acquire the video to be processed in real time. The image acquisition device 110 can transmit the acquired video to be processed to the processor 102, which then performs video processing.

[0036] The video to be processed may include any number of video frames. In one embodiment, the acquired video to be processed may include 128 video frames. The 128 video frames may be divided into any number of frame sequences, each frame sequence including one or more video frames. For example, the 128 video frames are divided into 16 frame sequences, the first frame sequence Clip0 may include video frames from 1 to 8, the second frame sequence Clip1 may include video frames from 9 to 16, and so on, the 16th frame sequence Clip15 may include video frames from 121 to 128. It will be understood that the current way of dividing the frame sequence is merely exemplary, and the number of video frames included in different frame sequences may be the same or different, and this application does not limit this. In addition, optionally, in the case where each frame sequence includes multiple video frames, the video frames included in the first frame sequence and the video frames included in the second frame sequence in any two adjacent frame sequences may be completely different or partially the same.

[0037] For each frame sequence, the target image features corresponding to that frame sequence can be obtained. The target image features corresponding to any frame sequence include the target image features corresponding to one or more video frames in the corresponding frame sequence. The target image features corresponding to any video frame can be solely the features of the region containing the target object in the current video frame, or they can be the features of the target object and the pixel region surrounding the target object. For example, they can include the features of the target object and the pixel region within 10 pixels surrounding the target object. Of course, the target image features can also include the features of the entire pixel region in the current video frame. The target object can be any object, including but not limited to pedestrians, animals, vehicles, etc. For example, the video to be processed can be a road surveillance video, and the target object can be a vehicle passing through the video. Alternatively, the target object can be a pedestrian in the road surveillance video. In embodiments of the present application, the target object can be a class of objects or a specific object. For example, if the target object is a pedestrian, then every pedestrian A, pedestrian B, pedestrian C, etc. appearing in the video to be processed is a target object. Alternatively, the target object can be a specific object. For example, in the embodiment described above where the target object is a pedestrian, the target object can be pedestrian B. In one embodiment, the target image features may include image edge, color, space and other information, which may be represented by a high-dimensional tensor. Any suitable feature extraction model may be used to extract the target image features corresponding to each frame sequence. Exemplarily and non-restrictively, the target image features of each frame sequence may be extracted by a transformer backbone network (Transformer Backbone). Optionally, in the process of extracting the target image features of the current frame sequence, the timing information may be obtained by interacting with the feature information of the previous frame sequence (such as the memory token features described below) through a cross-feature attention mechanism, thereby further improving the understanding of the entire video timing.

[0038] Step S220: Execute a frame sequence processing operation for any current frame sequence in the video to be processed. The frame sequence processing operation may include the following steps S221, S222, S223, S224, S225, S226, and S227. Figure 3 A schematic diagram of a frame sequence processing operation according to an embodiment of the present application is shown. Figure 2 and Figure 3 Describes how frame sequence processing operations are implemented.

[0039] Step S221 , performing target detection based on the target image features corresponding to the current frame sequence, and obtaining an initial target detection result corresponding to the current frame sequence, wherein the initial target detection result includes initial position information of the target object in each video frame in the corresponding frame sequence.

[0040] Exemplarily, the current frame sequence may be the second frame sequence Clip1, which contains the 9th to 16th frames of video. Each video frame corresponds to its own target image feature. Any existing or future target detection model may be used to perform target detection based on the target image features corresponding to the current frame sequence to obtain an initial target detection result F1 corresponding to the current frame sequence. Exemplarily, the target detection model may include but is not limited to a region-based convolutional neural network (RCNN), a fast region convolutional neural network (FasterRCNN), a single shot multi-scale target detector (SSD), a single-stage classic detector (YOLO), or a multi-view detector. Figure 3 Position embedding transformation for multi-view 3D object detection (PETR), etc. The initial target detection result F1 may include the position information of at least one initial target detection frame (referred to herein as "initial detection frame"). Each target detection frame described herein is a bounding box containing the target object, which may optionally be a rectangular frame. Of course, the shape of the target detection frame may also be other suitable shapes, such as a circle, a triangle, etc. The same target object may correspond to one or more target detection frames. It should be noted that the initial target detection result corresponding to each frame sequence may include the initial target detection result corresponding to each video frame in the frame sequence. Exemplarily, each target detection frame included in the initial target detection result corresponding to each frame sequence may carry corresponding tag information to indicate the video frame from which it originates, for example, the frame number of the video frame may be used as the tag information. In this way, in the initial target detection results corresponding to the same frame sequence, the tag information of each target detection frame can be used to identify which video frame it originates from. It can be understood that, similarly, for the final target detection result to be described below, each target detection frame contained therein may also carry label information related to the source video frame.

[0041] Reference Figure 3 , you can use a target detection model such as the YOLOX network to perform target detection based on the target image feature F-Clip0 corresponding to the first frame sequence Clip0 to obtain the initial target detection result F0 of the frame sequence. Figure 3The initial target detection result F0 shown only shows the initial target detection result corresponding to a single video frame. The initial target detection result corresponding to the single video frame includes the position information of the target detection frames corresponding to the five target objects. Assuming that the eight video frames in the first frame sequence Clip0 each correspond to five target detection frames, the initial target detection result corresponding to the first frame sequence Clip0 may include the position information of 40 target detection frames in total. Exemplarily, the initial target detection result corresponding to each frame sequence may also include the confidence level corresponding to each target detection frame. The confidence level may be used to indicate the probability that the corresponding target detection frame contains the target object. The greater the confidence level, the greater the possibility that the corresponding target detection frame contains the target object, which means that the target detection frame is more credible.

[0042] Step S222 : Position-encode at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature.

[0043] Exemplarily, all initial position information in the initial target detection results corresponding to the current frame sequence can be made to participate in position encoding. Exemplarily, the initial position information contained in the initial target detection results corresponding to the current frame sequence can also be filtered, so that part of the filtered initial position information can participate in position encoding. In one embodiment, filtering the initial position information can include but is not limited to: excluding the position information corresponding to the target detection frame whose confidence in the initial position information does not meet the requirements (i.e., the initial position information corresponding to the initial target detection frame); if multiple target detection frames contain the same target object, the position information corresponding to any one of the target detection frames can be retained.

[0044] By way of example and not limitation, position coding may include any one of conditional position coding, learnable absolute position coding, sine-cosine function coding, and relative position coding. In one embodiment, the sine-cosine function may be used to positionally encode at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain its corresponding first position coding feature. The first positional encoding feature may also be referred to as a first positional embedding feature. The first positional encoding feature may be a feature of dimension N1×C, which contains N1 feature vectors of length C.

[0045] Step S223: Obtain image embedding features corresponding to at least part of the position information in the initial object detection result corresponding to the current frame sequence.

[0046] Exemplarily, step S223 may include: determining an image embedding (token embedding) feature corresponding to at least part of the position information in the initial target detection result based on image information in the current frame sequence corresponding to at least part of the position information in the initial target detection result.

[0047] Exemplarily, the image information corresponding to the initial position information of any initial detection frame (i.e., the image information corresponding to the initial detection frame) can be obtained based on the target image features corresponding to the current frame sequence. As described above, at least part of the position information in the initial target detection result is the initial position information corresponding to at least part of the initial detection frame in the initial target detection result. For any initial detection frame, the local image features at the position of the initial detection frame can be extracted from the target image features corresponding to the current frame sequence as the image information corresponding to the initial detection frame. For example, any initial detection frame can be mapped to the target image features corresponding to the current frame sequence, for example, specifically mapped to the target image features corresponding to the source video frame of the initial detection frame, and then the local image features at the position of the initial detection frame are obtained as the image information corresponding to the initial detection frame. In this case, the local image features corresponding to any initial detection frame can be flattened into the form of sequence features to obtain image embedding features. The dimension of the image embedding feature is consistent with the first position encoding feature, and can be, for example, N1×C in size.

[0048] Exemplarily, the image information corresponding to the initial position information of any initial detection frame can be obtained based on the current frame sequence. For example, for any initial detection frame, the image block at the position of the initial detection frame can be extracted from its source video frame as the image information corresponding to the initial detection frame. Subsequently, feature extraction can be performed on the image information corresponding to the initial detection frame to obtain the local image features corresponding to the initial detection frame. Subsequently, the local image features corresponding to any initial detection frame can be flattened into a sequence feature form to obtain an image embedding feature.

[0049] Step S224: Fusing the first position coding feature with the image embedding feature to obtain a current query feature corresponding to the current frame sequence.

[0050] For example, the first position encoding feature and the image embedding feature can be added element by element to obtain the current query feature (such as Figure 3 Proposal Q shown in FIG. 1 ). For example, during position encoding, the initial position information of each initial detection frame can be normalized. Subsequently, the normalized result can be mapped to the same dimension as the image embedding feature through a linear layer to obtain a first position encoding feature, so that the first position encoding feature can be element-wise added to the image embedding feature.

[0051] Step S225: Generate a target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, wherein the current query feature, the updated query feature and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object.

[0052] For example, the previous frame sequence refers to a frame sequence that appears earlier than the current frame sequence on the time axis (e.g., captured earlier than the current frame sequence). For example, if the current frame sequence is the second frame sequence Clip1, the previous frame sequence is the first frame sequence Clip0. The appearance time of any frame sequence can be represented by the earliest appearance time of each video frame in the frame sequence.

[0053] For the convenience of description, "at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence" used in step S225 is referred to as the prior query feature. Exemplarily, the dimension of the current query (query) feature may be N1×C, and the dimension of the prior query feature may be N2×C, and the two may be combined to obtain a target query feature of dimension N×C, that is, N=N1+N2. N1, N2, and N can all be any integer greater than 0. The current query feature includes N1 feature vectors of length C, which correspond one-to-one to N1 potential target objects. The prior query feature includes N2 feature vectors of length C, which correspond one-to-one to N2 potential target objects. The target query feature includes N feature vectors of length C, which correspond one-to-one to N potential target objects. As Figure 3 As shown, the prior query feature can be represented by SeqQ, and the target query feature can be represented by Q.

[0054] For example, for the first frame sequence (e.g., Clip0), the updated query feature corresponding to the previous frame sequence can be the initialization query feature. The initialization query feature can be preset and can be optimized along with the parameters of the video processing model during the training process of the video processing model. The initialization query feature can be a feature with a dimension of N3×C, where N3 can be set to any size as needed. For the first frame sequence, all initialization query features can be used as prior query features, that is, N3=N2. The initialization query feature is optional. For the first frame sequence (e.g., Clip0), the updated query feature corresponding to the previous frame sequence can be 0 (i.e., none). From the above description, it can be understood that the channel number C of the initialization query feature, current query feature, prior query feature, update query feature, and target query feature remains consistent. For any frame sequence other than the first frame sequence (e.g., Clip0), the prior query feature SeqQ used to generate the target query feature can come from the updated query feature corresponding to the previous frame sequence. As the frame sequence is processed from forward to backward, the corresponding updated query feature can be continuously updated to incorporate the feature information of each processed frame sequence. Thus, the feature information of the previous frame sequence can be integrated into the updated query feature as a priori, and the current frame sequence can be detected based on at least the updated query feature containing the priori knowledge to determine the position of the target object in the current frame sequence.

[0055] Step S226: Decoding is performed based on the target image features and the target query features corresponding to the current frame sequence to obtain updated query features corresponding to the current frame sequence. The decoding may include a cross-attention operation.

[0056] For example, the decoding module in the video processing model can be used to decode the target image features (e.g. Figure 3The F-Clip0 or F-Clip1 shown in the figure and the target query feature Q are decoded to obtain the updated query feature Q' corresponding to the current frame sequence. Exemplarily, the video processing model can be at least part of the network structure in the video instance segmentation model (VIS) based on the transformer structure. The decoding module can be at least part of the network structure in any decoder module, and the above-mentioned decoder module can be, for example, a deformable detection decoder (Deformable DETR). Exemplarily, the decoding module can include the remaining network structure in the decoder module except the head module (head). Those skilled in the art will understand that the head module can include one or more of a detection head for outputting position information corresponding to the target detection frame, a classification head for outputting classification information corresponding to the target detection frame, a segmentation head for outputting mask information corresponding to the target detection frame, etc. The decoding module can include one or more attention operation layers and one or more multilayer perceptrons (MLPs).

[0057] Step S226 can be regarded as an operation of searching and tracking target object information. Through attention operation, information of each potential target object can be continuously tracked to predict whether the potential target object actually exists and further predict its exact location if it exists.

[0058] Step S227 , based on the updated query features corresponding to the current frame sequence, determining the final target detection result corresponding to the current frame sequence, wherein the final target detection result includes final position information of the target object in each video frame in the corresponding frame sequence.

[0059] For example, the updated query feature Q' corresponding to the frame sequence Clip0 can be input into the subsequent header module to obtain the final target detection result R-Clip0 corresponding to the frame sequence Clip0. Similarly, the updated query feature Q' corresponding to the frame sequence Clip1 can be input into the subsequent header module to obtain the final target detection result R-Clip1 corresponding to the frame sequence Clip1.

[0060] According to the video processing method of the embodiment of the present application, position encoding is performed based on at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature, and then the current query feature corresponding to the current frame sequence is obtained based on the first position encoding feature. The method further generates a target query feature based on at least part of the feature vector in the updated query feature corresponding to the previous frame sequence and the current query feature, and performs a query based on the generated target query feature to obtain the final target detection result of the current frame sequence. The method can integrate the target detection information of the previous frame sequence as a priori into the query feature of the current frame sequence, so that the temporal relationship of the video can be modeled, which helps to achieve continuous search and tracking of the target object in the video frame, and thus can effectively and accurately detect and identify the video instance sequence. In addition, the method can also perform target detection through the target image features corresponding to each of the multiple frame sequences, thereby ensuring a relatively accurate single-frame target object detection (and instance segmentation) effect. Therefore, the above scheme allows for relatively accurate single-frame target detection (and instance segmentation) of the video frame, while also making the final target detection result obtained continuous and consistent in time sequence, which helps to obtain relatively accurate position information of the target object in both the temporal and spatial dimensions.

[0061] Exemplarily, the video processing method according to the embodiment of the present application can be implemented in a device, apparatus or system having a memory and a processor.

[0062] The video processing method according to the embodiment of the present application can be deployed at an image acquisition end, for example, can be deployed at a personal terminal or a server end with an image acquisition function.

[0063] Alternatively, the video processing method according to the embodiment of the present application can also be deployed in a distributed manner on the server side (or cloud) and the personal terminal. For example, the video to be processed can be obtained on the client side, and the client side transmits the obtained video to the server side (or cloud), and the server side (or cloud) performs the video processing.

[0064] Exemplarily, for the first frame sequence in the video to be processed, the updated query feature corresponding to the previous frame sequence is the initialization query feature.

[0065] In one embodiment, the initialization query feature Q can be generated in advance pre Initialize query feature Q pre It can be represented as a feature of dimension N3×C, where N3 can be any integer greater than 0. For example, the query feature Q is initialized preThe dimension may be 300×C. The initialization query feature may be pre-stored in a local storage space or a cloud storage space of an apparatus (eg, the electronic device 100 ) for implementing the video processing method according to an embodiment of the present application.

[0066] For example, for any current frame sequence, at least some of the feature vectors in the updated query features corresponding to the previous frame sequence may include the feature vectors corresponding to the initialization query features in the updated query features corresponding to the previous frame sequence. That is, during video processing, the feature vectors corresponding to the initialization query features may be retained as at least part of the prior query features to participate in instance queries for the current frame sequence.

[0067] According to the above technical solution, the target query feature corresponding to the first frame sequence can be obtained in combination with the initialization query feature, so that additional prior information can be introduced into the target query feature corresponding to the first frame sequence. Due to the generation principle of the target query features of each frame sequence described above, it can be known that the additional prior information can also be introduced into subsequent frame sequences, which helps to detect the target object in each video frame more comprehensively and accurately.

[0068] Exemplarily, the initial position information is used to indicate the predicted position of the initial detection frame where the target object is located, and the final position information is used to indicate the predicted position of the final detection frame where the target object is located. The initial target detection result may also include a confidence level corresponding to each initial detection frame, and the final target detection result may also include a confidence level corresponding to each final detection frame. Exemplarily, before position encoding at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain the first position encoding feature, the frame sequence processing operation may also include: selecting the initial position information corresponding to the initial detection box in the initial target detection result corresponding to the current frame sequence, whose confidence is greater than or equal to the first confidence threshold, as at least part of the information in the initial target detection result corresponding to the current frame sequence; and / or, before generating the target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, the frame sequence processing operation also includes: selecting the final detection box in the final target detection result corresponding to the previous frame sequence, whose confidence is less than the second confidence threshold, and using the feature vectors in the updated query feature corresponding to the previous frame sequence except for the specific feature vector as at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence, wherein the specific feature vector is the feature vector corresponding to the selected final detection box.

[0069] As described above, in one embodiment, the initial target detection result may include initial position information for the initial detection frame where the target object is located. This position information indicates the predicted position of the initial detection frame. The initial target detection result may also include the confidence level corresponding to each initial detection frame. The final target detection result may include final position information for the final detection frame where the target object is located. This position information indicates the predicted position of the final detection frame. The final target detection result may also include the confidence level corresponding to each final detection frame. Taking the initial detection frame as an example, the initial position information may include one or more of the following information about the initial detection frame: corner coordinates of one or more corner points; center coordinates; width information; and height information. If the initial position information includes width and / or height information, the initial position information may further include corner coordinates and / or center coordinates of at least one corner point. The confidence level of the initial detection frame may be represented by any numerical value, for example, the confidence level may range from 0 to 1. As described above, a confidence level closer to 1 indicates a more accurate detection of the target object by the initial detection frame. Similarly, the meaning of the final position information and confidence level of the final detection frame can be understood by reference.

[0070] The first confidence threshold corresponding to the initial detection frame and the second confidence threshold corresponding to the final detection frame can be pre-set. The first confidence threshold and the second confidence threshold can be any value between 0 and 1, and the two can be the same or different. Exemplarily, the first confidence threshold can be 0.6 and the second confidence threshold can be 0.7. For example, the initial position information corresponding to the initial detection frame with a confidence greater than or equal to the first confidence threshold of 0.6 in the initial target detection result corresponding to the current frame sequence can be selected as at least part of the information in the initial target detection result corresponding to the current frame sequence. For example, the final detection frame with a confidence less than the second confidence threshold of 0.7 in the final target detection result corresponding to the previous frame sequence can be selected, and the feature vectors corresponding to these selected final detection frames are excluded from the updated query feature Q' corresponding to the previous frame sequence, and the remaining feature vectors are used as at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence, that is, the prior query feature SeqQ.

[0071] In one example, the initial target detection result corresponding to the frame sequence Clip0 includes 20 initial detection frames, and the corresponding current query feature ProposalQ may include 20 feature vectors (which may be referred to as queries), and the prior query feature SeqQ may be an initialization query feature, which may include, for example, 300 queries. In this way, the target query feature Q corresponding to the frame sequence Clip0 may include a total of 320 queries. Correspondingly, the updated query feature Q' corresponding to the frame sequence Clip0 may also include 320 queries. For the updated query feature Q' corresponding to the frame sequence Clip0, among the 20 feature vectors corresponding to the current query feature ProposalQ, the 12 queries corresponding to the 12 final detection frames whose confidence is lower than the second confidence threshold are excluded, and the remaining 8 queries, together with the 300 queries corresponding to the initialization query feature, form a new 308 queries. These 308 queries can be used as the prior query feature SeqQ corresponding to the next frame sequence Clip1. Assuming that the initial target detection result corresponding to the frame sequence Clip1 includes 10 initial detection frames, the target query feature Q including 318 queries can be obtained. Correspondingly, the updated query feature Q' corresponding to the frame sequence Clip1 can also include 318 queries. For the updated query feature Q' corresponding to the frame sequence Clip1, among the 10 feature vectors corresponding to the current query feature ProposalQ, the 6 queries corresponding to the 6 final detection frames with confidence lower than the second confidence threshold are excluded, and the remaining 4 queries are combined with the other 308 queries to form a new 312 queries as the prior query feature SeqQ of the frame sequence Clip2. And so on. Based on this example, it can also help to understand the above-mentioned scheme of "always retaining the feature vector corresponding to the initialization query feature as at least part of the prior query feature to participate in the instance query of the current frame sequence."

[0072] According to the above technical solution, the target detection frames in the initial target detection results and the final target detection results are screened respectively by using a pre-set confidence threshold, so as to avoid the target detection frames with low confidence from participating in the instance query of the current or next frame sequence. This can make the final position information obtained more accurate and effectively reduce the amount of calculation.

[0073] Exemplarily, in the case where each frame sequence includes multiple video frames, in any two adjacent frame sequences, the video frames included in the first frame sequence are partially identical to the video frames included in the second frame sequence.

[0074] In one embodiment, for any two adjacent frame sequences in a plurality of frame sequences, such as the second frame sequence Clip1 and the third frame sequence Clip2, the plurality of video frames included in the second frame sequence Clip1 and the plurality of video frames included in the third frame sequence Clip2 may be completely different or partially the same. For example, the second frame sequence Clip1 may include video frames from the 9th to the 16th frames, and the third frame sequence Clip2 may include video frames from the 17th to the 24th frames. In another embodiment, the second frame sequence Clip1 may include video frames from the 9th to the 16th frames, and the third frame sequence Clip2 may include video frames from the 14th to the 21st frames. That is, the second frame sequence Clip1 and the third frame sequence Clip2 overlap at video frames from the 14th to the 16th frames.

[0075] The above technical solution allows for flexible division of frame sequences, allowing for a certain degree of overlap between adjacent frames on the time axis. This solution, a sliding-frame video processing solution, enables sliding-frame video feature extraction, ensuring better spatiotemporal continuity in the final object detection results, thereby improving object detection accuracy.

[0076] Exemplarily, obtaining target image features corresponding to multiple frame sequences of a video to be processed may include: for any current frame sequence in the video to be processed, performing feature extraction on each video frame in the current frame sequence to obtain initial image features corresponding to the current frame sequence, the initial image features corresponding to the current frame sequence include initial image features corresponding to one or more video frames in the current frame sequence; fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain memory token features corresponding to the current frame sequence; fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the current frame sequence to obtain target image features corresponding to the current frame sequence.

[0077] In one embodiment, the initial image features corresponding to the video frames may be only the features of the area containing the target object in the video frames corresponding to the current frame sequence, or may be the features of the target object and the pixel area near the target object. For example, the features of the target object and the pixel area within 10 pixels near the target object may be included. Of course, the initial image features may also include the features of all pixel areas in the video frames corresponding to the current frame sequence. In one embodiment, the initial image features may include information such as the edge, color, and space of the image, which may be represented by a high-dimensional tensor. The initial image features may be extracted by any feature extraction model, such as an encoder model.

[0078] For the video to be processed, a temporal memory can be formed by using memory token features as long-term memory storage units to memorize and store feature information from a series of frame sequences. Previous memories can then be fused with the initial image features corresponding to the current frame sequence to form new memory token features. Memory token features can also be represented using high-dimensional tensors. By way of example and not limitation, the dimensions of the memory token features and the image features corresponding to any video frame in any frame sequence (including the initial image features and / or the target image features) can be the same or different. Preferably, the dimensions of the memory token features are smaller than the dimensions of the image features corresponding to any video frame. For example, assume that the initial image features can be represented as a three-dimensional tensor with dimensions H1×W1×C1, where C1 is the number of channels and H1 and W1 are the height and width of the feature map for each channel. Memory token features can also be represented as a three-dimensional tensor with dimensions H2×W2×C2, where each dimension has a similar meaning to the initial image features. Preferably, C1 and C2 are equal, H2 is smaller than H1, and W1 is smaller than W2. For example, for the first frame sequence Clip0, an initialization memory token feature (which may be referred to as a first initialization memory token feature) may be used as the memory token feature corresponding to the previous frame sequence. For example, the initialization memory token feature may establish a priori model based on initial perception of information such as the position and / or shape of the target object. Then, the perception is automatically updated based on the input video frames to obtain memory token features corresponding to each frame sequence, so that the updated memory token features are responsible for the current video (e.g., the video to be processed).

[0079] Exemplarily, the fusion method of the initial image features corresponding to the current frame sequence and the memory token features corresponding to the previous frame sequence in the video to be processed can be the sum of each feature, or the multiplication of each feature, or the fusion can be achieved through attention mechanism operation, similarity calculation, etc.

[0080] According to the above technical solution, the memory token feature can be used to store and transmit the feature information of each frame sequence, achieving a long-term memory effect of the feature information. In addition, the solution can fuse the initial image features of the current frame sequence with the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence. The target image features thus obtained also incorporate certain temporal feature information.

[0081] Exemplarily, fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain the memory token features corresponding to the current frame sequence may include: position encoding the initial image features corresponding to the current frame sequence to obtain a second position encoding feature, the second position encoding feature having the same dimension as the initial image features corresponding to the current frame sequence; merging the second position encoding feature with the initial image features corresponding to the current frame sequence to obtain a merged feature; performing an attention mechanism operation on the merged feature and the memory token features corresponding to the previous frame sequence to obtain the memory token features corresponding to the current frame sequence; fusing the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence, may include: performing an attention mechanism operation on the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence.

[0082] In one embodiment, the step of fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain the memory token features corresponding to the current frame sequence may include the following steps: position encoding the initial image features corresponding to the current frame sequence to obtain a second position encoding feature, wherein the second position encoding feature has the same dimension as the initial image features corresponding to the current frame sequence; after obtaining the second position encoding feature, merging the second position encoding feature with the initial image features corresponding to the current frame sequence to obtain a merged feature; and performing an attention mechanism operation on the merged feature and the memory token features corresponding to the previous frame sequence to obtain a memory token feature corresponding to the current frame sequence.

[0083] Position encoding encodes each region in each video frame in the current frame sequence. When extracting initial image features from any current video frame, the current video frame can be stretched and represented as a tensor with 1 column and d rows. By encoding each region in the current video frame, the position of each region can be identified after the image is stretched. For example, for a 4×4 video frame, the stretched video frame can be represented as a 1×16-dimensional tensor, where the 16 elements contained therein can correspond one-to-one to the pixel values ​​of the 16 pixels in the video frame. For each of the 16 positions in the image, a second position encoding feature can be obtained, which can also be referred to as a second position embedding feature. For example, the position of a 1×1 region can be labeled as 0, the position of a 1×2 region as 1, and so on, and the position of a 4×4 region as 15. In one embodiment, the position labels can be continuous, such as 1, 2, 3, 4, 5, ..., 16. Alternatively, the position labels can be discontinuous, such as 1, 3, 4, 6, 9, ..., 23.

[0084] For example, the position encoding method can be implemented through self-learning of a neural network model. The neural network model can be a back propagation (BP) neural network, a Hopfield network, an adaptive resonance theory (ART) network, a Kohonen network, etc. Alternatively, the position encoding method can use the cosine position encoding formula to initialize the position of each region. This solution encodes the position of each region using the cosine position encoding formula, and the encoded value is relatively stable and the result is relatively reliable.

[0085] Exemplarily, the merging of the second position coding feature and the initial image feature may be element-wise addition. In one embodiment, the initial image feature is f t , the position corresponding to the initial image feature is pos1, then the merged feature f t ′=f t +pos1. The merged feature f t ′The memory token feature m corresponding to the previous frame sequence t-1 Performing attention mechanism operation, we can obtain the memory token feature m corresponding to the current frame sequence t The above-mentioned attention mechanism operation can specifically be a cross-attention mechanism operation. The following describes an exemplary implementation of the attention mechanism operation adopted in this application.

[0086] For example, the memory token feature m corresponding to the previous frame sequence can be used t-1 The feature information q1 and the initial image feature f corresponding to the current frame sequence t The feature information k1 and v1 are used for attention mechanism operation, and the memory token feature m corresponding to the current frame sequence is obtained. t =Attn(k1,q1,v1)+q1. In one embodiment, k1=f t ′,q1=m t-1 , v1=f t . You can first calculate the similarity of k1 and q1 to get a similarity matrix. For example, if k1 and q1 are both 1×16 dimensional tensors, you can get a 16×16 similarity matrix. Then, you can multiply the similarity matrix with v1, and finally add the multiplication result to q1 to get m t .

[0087] According to the above technical solution, by interacting the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence and utilizing the cross-attention mechanism, the memory token features can be efficiently and systematically memorized and stored, which helps to achieve temporal memory.

[0088] Exemplarily, the step of fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence may include the following steps: performing an attention mechanism operation on the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence.

[0089] For example, the memory token feature m corresponding to the current frame sequence can be used t The feature information k2 and v2 and the initial image feature f corresponding to the current frame sequence t The feature information q2 is used for attention mechanism operation, and the memory token feature corresponding to the current frame sequence is calculated. In one embodiment, q2=k1=f t ′. k2=v2=m t . You can first calculate the similarity between k2 and q2 to get the similarity matrix. Then, multiply the similarity matrix with v2, and finally add the multiplication result to q2 to get

[0090] According to the above technical solution, the attention mechanism matches the initial image features with the memory token features corresponding to the current frame sequence, and uses attention-based interaction to quickly and accurately read local feature information. This solution can use memory information to enhance the features of the current frame sequence and significantly improve the consistency of the timing.

[0091] Exemplarily, the final position information is used to indicate the predicted position of the final detection frame where the target object is located. After determining the final target detection result of the current frame sequence based on the updated query feature corresponding to the current frame sequence, the frame sequence processing operation may also include: based on the final target detection result of the current frame sequence, mapping at least part of the final detection frame to the target image feature corresponding to the current frame sequence, and obtaining local image features corresponding to at least part of the final detection frames; using the local image feature corresponding to any final detection frame as a convolution kernel, convolving the target image feature corresponding to the current frame sequence, and obtaining mask information corresponding to the final detection frame, the mask information is used to indicate the position of the mask of the target object contained in the corresponding final detection frame.

[0092] In one embodiment, after obtaining the final object detection results for the current frame sequence, the pixel positions of all or part of the pixels in the final detection frames contained in the final object detection results can be mapped to the target image features corresponding to the current frame sequence, thereby obtaining the local image features corresponding to each of the final detection frames. For example, if there are five final detection frames, then the local image features corresponding to each of the five final detection frames can be obtained. This means that the number of local image features is also five.

[0093] By using any of these five sets of local image features as a convolution kernel and convolving it with the target image features corresponding to the current frame sequence, we can obtain mask information corresponding to the final detection box. This convolution operation can be implemented in the segmentation head described above. The mask information can be presented as a heat map. In the heat map, each pixel within the target object's mask can be highlighted. The mask information can be viewed as the instance segmentation result obtained by performing instance segmentation on the target object.

[0094] According to the above technical solution, the local image features corresponding to any final detection frame are used as convolution kernels to convolve the target image features corresponding to the current frame sequence, obtaining mask information corresponding to the final detection frame. This method can achieve more accurate instance segmentation and improve the accuracy of the obtained target object location.

[0095] Exemplarily, target detection is performed based on the target image features corresponding to the current frame sequence, and the initial target detection result corresponding to the current frame sequence is obtained by the target detection module in the video processing model. Based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, the target query feature is generated by the decoding module in the video processing model. The video processing model can be trained in the following way: obtaining the labeled target detection results and target image features corresponding to multiple frame sequences of the sample video one by one, the labeled target detection results including the labeled position information of the target object in each video frame in the corresponding frame sequence; for any current frame sequence in the sample video, using the video processing model to perform frame sequence processing operations to obtain the predicted target detection result corresponding to the current frame sequence; based on the predicted target detection results and labeled target detection results corresponding to each of the multiple frame sequences in the sample video, the prediction loss is calculated; and the parameters in the video processing model are optimized based on the prediction loss.

[0096] In one embodiment, those skilled in the art will understand the implementation method for obtaining target image features corresponding to multiple frame sequences of a sample video by referring to the method for obtaining target image features corresponding to multiple frame sequences of a video to be processed in the previous embodiment. For the sake of brevity, this description will not be repeated here. It will be understood that the annotated target detection results may include the annotated location information of the target object in each video in the corresponding frame sequence. For any current frame sequence in a sample video, a predicted target detection result for the current frame sequence can be obtained using a video processing model. It will be understood that the predicted target detection result corresponding to the current frame sequence in the sample video is the final target detection result corresponding to the current frame sequence. This video processing model can have the same network structure as the video processing model used in step S220, but the parameters may be different. After training the parameters of the initial video processing model, the resulting video processing model is used in step S220. The predicted target detection results and annotated target detection results corresponding to each of the multiple frame sequences in the sample video can be substituted into a first preset loss function to calculate the loss and obtain a first predicted loss. In this embodiment, annotated mask information for each video frame in the multiple frame sequences of the sample video can also be obtained in advance. Based on the position information corresponding to each target detection frame in the predicted target detection results obtained above, the predicted mask information corresponding to each target detection frame can be obtained. The method for determining the predicted mask information can be understood by referring to the method for determining the mask information of the current frame sequence in the video to be processed. After substituting the labeled mask information and the predicted mask information into the second preset loss function, the second prediction loss can be determined. The first prediction loss can be used as the prediction loss of the video processing model (which can be called the total prediction loss) for optimization, or the second prediction loss and the second prediction loss can be combined to obtain the prediction loss of the video processing model for optimization. By way of example and not limitation, the first preset loss function can be a mean squared error loss function, a squared loss function, etc., and the second preset loss function can be a binary cross entropy loss function (BCE), etc. The parameters in the initial video processing model are optimized using backpropagation and gradient descent algorithms. The parameter optimization can be iteratively performed until the video processing model reaches a convergence state. After training is completed, the obtained video processing model can be used for subsequent video processing. This stage can be called the inference or testing stage of the model.

[0097] In addition, exemplarily, during the training process, noise can be added to the labeled target detection results corresponding to any one or more frame sequences of the sample video to obtain the labeled position information generated based on the noise (i.e., the noise detection box) for training. This can improve the convergence efficiency of the video processing model to improve the performance of the video processing model.

[0098] According to the above technical solution, by obtaining the labeled target detection results of the sample video and multiple frame sequences of the sample video in one-to-one correspondence, the video processing model is trained, thereby optimizing the parameters of the video processing model and thereby improving the performance of the video processing model.

[0099] Exemplarily, optimizing parameters in the video processing model based on the prediction loss may include optimizing the parameters in the video processing model and the initialization query features together based on the prediction loss.

[0100] In one embodiment, based on the prediction loss of the video processing model obtained above, the parameters (including weights and / or biases, etc.) in the video processing model and the initialized query features can be adjusted together through back propagation and gradient descent algorithms, that is, optimized, so that the similarity between the predicted target detection results output by the video processing model and the labeled target detection results is improved.

[0101] According to the above technical solution, by optimizing the parameters in the video processing model and the initialization query features based on the prediction loss, the performance of the video processing model can be further improved to ensure the accuracy of the final target detection results.

[0102] According to another aspect of the present application, a video processing device is provided. Figure 4 FIG4 shows a schematic block diagram of a video processing device 400 according to an embodiment of the present application.

[0103] like Figure 4 As shown, the video processing device 400 according to the embodiment of the present application includes an acquisition module 410 and a processing module 420. The processing module 420 may include a detection submodule 421, an encoding submodule 422, an acquisition submodule 423, a fusion submodule 424, a generation submodule 425, a decoding submodule 426 and a determination submodule 427. Each module may respectively perform the above Figure 2 The following only describes the main functions of the components of the video processing device 400, and omits the details already described above.

[0104] The acquisition module 410 is used to obtain target image features corresponding to multiple frame sequences of the video to be processed. Each frame sequence in the multiple frame sequences includes one or more video frames. The target image features corresponding to any frame sequence include target image features corresponding to one or more video frames in the corresponding frame sequence. The acquisition module 410 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement the above.

[0105] The processing module 420 is used to perform frame sequence processing operations on any current frame sequence in the video to be processed. The processing module 420 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement the above.

[0106] Specifically, the detection submodule 421 is used to perform target detection based on the target image features corresponding to the current frame sequence, and obtain the initial target detection result corresponding to the current frame sequence, wherein the initial target detection result includes the initial position information of the target object in each video frame in the corresponding frame sequence.

[0107] The encoding submodule 422 is configured to perform position encoding on at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature.

[0108] The acquisition submodule 423 is used to acquire image embedding features corresponding to at least part of the position information in the initial object detection result corresponding to the current frame sequence.

[0109] The fusion submodule 424 is used to fuse the first position coding feature with the image embedding feature to obtain the current query feature corresponding to the current frame sequence.

[0110] The generation submodule 425 is used to generate a target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, wherein the current query feature, the updated query feature and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object.

[0111] The decoding submodule 426 is configured to perform decoding based on the target image features and the target query features corresponding to the current frame sequence to obtain updated query features corresponding to the current frame sequence.

[0112] The determination submodule 427 is used to determine the final target detection result corresponding to the current frame sequence based on the updated query feature corresponding to the current frame sequence, wherein the final target detection result includes the final position information of the target object in each video frame in the corresponding frame sequence.

[0113] Figure 5 FIG2 is a schematic block diagram of an electronic device 500 according to an embodiment of the present application. The electronic device 500 includes a memory 510 and a processor 520 .

[0114] The memory 510 stores computer program instructions for implementing corresponding steps in the video processing method according to the embodiment of the present application.

[0115] The processor 520 is configured to execute computer program instructions stored in the memory 510 to perform corresponding steps of the video processing method according to the embodiment of the present application.

[0116] In one embodiment, the computer program instructions are used by the processor 520 to perform the following steps when they are executed: obtaining target image features corresponding to each of multiple frame sequences of the video to be processed, each frame sequence in the multiple frame sequences includes one or more video frames, and the target image features corresponding to any frame sequence include target image features corresponding to one or more video frames in the corresponding frame sequence; for any current frame sequence in the video to be processed, performing the following frame sequence processing operations: performing target detection based on the target image features corresponding to the current frame sequence to obtain an initial target detection result corresponding to the current frame sequence; position encoding at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature; obtaining an image embedding feature corresponding to at least part of the position information in the initial target detection result corresponding to the current frame sequence; and embedding the first position encoding feature into the image embedding feature. A position coding feature is fused with an image embedding feature to obtain a current query feature corresponding to a current frame sequence; a target query feature is generated based on at least part of the feature vectors in the updated query feature corresponding to a previous frame sequence and the current query feature, wherein the current query feature, the updated query feature and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object; decoding is performed based on the target image feature and the target query feature corresponding to the current frame sequence to obtain the updated query feature corresponding to the current frame sequence; based on the updated query feature corresponding to the current frame sequence, a final target detection result corresponding to the current frame sequence is determined; wherein the initial target detection result includes initial position information of the target object in each video frame in the corresponding frame sequence, and the final target detection result includes final position information of the target object in each video frame in the corresponding frame sequence.

[0117] For example, the electronic device 500 may further include an image acquisition device 530. The image acquisition device 530 is used to acquire the video to be processed. The image acquisition device 530 is optional, and the electronic device 500 may also not include the image acquisition device 530. In this case, the processor 520 may obtain the video to be processed by other means, such as from an external device or from the memory 510.

[0118] In addition, according to an embodiment of the present application, a storage medium is provided on which program instructions are stored. When the program instructions are executed by a computer or processor, the storage medium is used to execute the corresponding steps of the video processing method according to the embodiment of the present application, and is used to implement the corresponding modules in the video processing device according to the embodiment of the present application. The storage medium may include, for example, a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0119] In one embodiment, when the program instructions are executed by a computer or processor, the computer or processor may implement the various functional modules of the video processing device according to the embodiment of the present application, and / or may execute the video processing method according to the embodiment of the present application.

[0120] In one embodiment, the program instructions are used to perform the following steps when running: obtaining target image features corresponding to each of a plurality of frame sequences of a video to be processed, each frame sequence in the plurality of frame sequences includes one or more video frames, and the target image features corresponding to any frame sequence include target image features corresponding to one or more video frames in the corresponding frame sequence; for any current frame sequence in the video to be processed, performing the following frame sequence processing operations: performing target detection based on the target image features corresponding to the current frame sequence to obtain an initial target detection result corresponding to the current frame sequence; position encoding at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature; obtaining an image embedding feature corresponding to at least part of the position information in the initial target detection result corresponding to the current frame sequence; encoding the first position encoding feature The features are fused with the image embedding features to obtain the current query features corresponding to the current frame sequence; based on at least part of the feature vectors in the updated query features corresponding to the previous frame sequence and the current query features, a target query feature is generated, wherein the current query feature, the updated query feature and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object; based on the target image features corresponding to the current frame sequence and the target query features, decoding is performed to obtain the updated query features corresponding to the current frame sequence; based on the updated query features corresponding to the current frame sequence, a final target detection result corresponding to the current frame sequence is determined; wherein the initial target detection result includes the initial position information of the target object in each video frame in the corresponding frame sequence, and the final target detection result includes the final position information of the target object in each video frame in the corresponding frame sequence.

[0121] In addition, according to an embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, and the computer program is used to execute the above-mentioned video processing method 200 when running.

[0122] Each module in the electronic device according to the embodiment of the present application can be implemented by running computer program instructions stored in a memory by a processor of an electronic device that implements video processing or video processing according to the embodiment of the present application, or can be implemented when computer instructions stored in a computer-readable storage medium of a computer program product according to the embodiment of the present application are executed by a computer.

[0123] In addition, according to an embodiment of the present application, a computer program is also provided, which is used to execute the above-mentioned video processing method 200 when running.

[0124] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0125] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not performed.

[0127] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0128] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various application aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features that are less than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0129] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0130] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0131] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the video processing device according to the embodiment of the present application. The application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0132] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0133] The above is merely a description of specific embodiments of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video processing method, comprising: Obtaining target image features corresponding to each of a plurality of frame sequences of a video to be processed, where each frame sequence in the plurality of frame sequences includes one or more video frames, and the target image features corresponding to any frame sequence include target image features corresponding to the one or more video frames in the corresponding frame sequence; For any current frame sequence in the video to be processed, perform the following frame sequence processing operations: Performing target detection based on target image features corresponding to the current frame sequence to obtain an initial target detection result corresponding to the current frame sequence; Performing position encoding on at least part of position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature; Obtaining an image embedding feature corresponding to at least part of the position information in the initial object detection result corresponding to the current frame sequence; fusing the first position encoding feature with the image embedding feature to obtain a current query feature corresponding to the current frame sequence; generating a target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, wherein the current query feature, the updated query feature, and the target query feature each include a feature vector corresponding one-to-one to at least one potential target object; Decoding the target image feature corresponding to the current frame sequence and the target query feature to obtain an updated query feature corresponding to the current frame sequence; Determining a final target detection result corresponding to the current frame sequence based on the updated query feature corresponding to the current frame sequence; The initial target detection result includes initial position information of the target object in each video frame in the corresponding frame sequence, and the final target detection result includes final position information of the target object in each video frame in the corresponding frame sequence.

2. The method according to claim 1, wherein The initial position information is used to indicate the predicted position of the initial detection frame where the target object is located, and the final position information is used to indicate the predicted position of the final detection frame where the target object is located. The initial target detection result also includes a confidence level corresponding to each initial detection frame, and the final target detection result also includes a confidence level corresponding to each final detection frame. Before position encoding at least part of the position information in the initial target detection result corresponding to the current frame sequence to obtain a first position encoding feature, the frame sequence processing operation further includes: selecting, in the initial target detection result corresponding to the current frame sequence, initial position information corresponding to the initial detection frame whose confidence is greater than or equal to a first confidence threshold, as at least part of the information in the initial target detection result corresponding to the current frame sequence; and / or, Before generating the target query feature based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, the frame sequence processing operation further includes: Select a final detection frame whose confidence is less than a second confidence threshold in the final target detection result corresponding to the previous frame sequence, and use the feature vectors in the updated query features corresponding to the previous frame sequence except for the specific feature vector as at least part of the feature vectors in the updated query features corresponding to the previous frame sequence, wherein the specific feature vector is the feature vector corresponding to the selected final detection frame.

3. The method according to claim 1, wherein In the case where each frame sequence includes multiple video frames, in any two adjacent frame sequences, the video frames included in the first frame sequence are partially identical to the video frames included in the second frame sequence.

4. The method according to any one of claims 1 to 3, wherein: The step of obtaining target image features corresponding to each of the plurality of frame sequences of the video to be processed includes: For any current frame sequence in the video to be processed, Performing feature extraction on each video frame in the current frame sequence to obtain initial image features corresponding to the current frame sequence, where the initial image features corresponding to the current frame sequence include initial image features corresponding to one or more video frames in the current frame sequence; Fusing the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain the memory token features corresponding to the current frame sequence; The initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence are fused to obtain the target image features corresponding to the current frame sequence.

5. The method according to claim 4, wherein: The fusing of the initial image features corresponding to the current frame sequence with the memory token features corresponding to the previous frame sequence in the video to be processed to obtain the memory token features corresponding to the current frame sequence includes: Performing position encoding on the initial image features corresponding to the current frame sequence to obtain a second position encoding feature, where the second position encoding feature has the same dimension as the initial image features corresponding to the current frame sequence; Merging the second position coding feature with the initial image feature corresponding to the current frame sequence to obtain a merged feature; Performing an attention mechanism operation on the merged feature and the memory token feature corresponding to the previous frame sequence to obtain a memory token feature corresponding to the current frame sequence; The fusing the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence includes: An attention mechanism operation is performed on the initial image features corresponding to the current frame sequence and the memory token features corresponding to the current frame sequence to obtain the target image features corresponding to the current frame sequence.

6. The method according to any one of claims 1 to 3, wherein: The final position information is used to indicate the predicted position of the final detection frame where the target object is located. After determining the final target detection result of the current frame sequence based on the updated query feature corresponding to the current frame sequence, the frame sequence processing operation further includes: Based on the final target detection result of the current frame sequence, mapping at least part of the final detection frame to the target image features corresponding to the current frame sequence, and obtaining local image features corresponding to each of the at least part of the final detection frame; The local image features corresponding to any final detection frame are used as the convolution kernel to convolve the target image features corresponding to the current frame sequence to obtain the mask information corresponding to the final detection frame. The mask information is used to indicate the position of the mask of the target object contained in the corresponding final detection frame.

7. The method according to any one of claims 1 to 3, wherein: The target detection is performed based on the target image features corresponding to the current frame sequence, and the initial target detection result corresponding to the current frame sequence is obtained by the target detection module in the video processing model. The target query feature is generated based on at least part of the feature vectors in the updated query feature corresponding to the previous frame sequence and the current query feature, and is generated by the decoding module in the video processing model. The video processing model is trained in the following way: Obtaining labeled target detection results and target image features corresponding to a plurality of frame sequences of a sample video, wherein the labeled target detection results include labeled position information of the target object in each video frame in the corresponding frame sequence; For any current frame sequence in the sample video, performing the frame sequence processing operation using the video processing model to obtain a predicted target detection result corresponding to the current frame sequence; Calculating prediction loss based on the predicted target detection results and the labeled target detection results corresponding to each of the plurality of frame sequences in the sample video; Parameters in the video processing model are optimized based on the prediction loss.

8. An electronic device comprising a processor and a memory, wherein: The memory stores computer program instructions, which are used by the processor to execute the video processing method according to any one of claims 1 to 7 when the processor is running the computer program instructions.

9. A storage medium having program instructions stored thereon, wherein: The program instructions are used to execute the video processing method according to any one of claims 1 to 7 when running.

10. A computer program product comprising a computer program, wherein: The computer program is used to execute the video processing method according to any one of claims 1 to 7 when running.

Images