Image processing method and apparatus, electronic device, and computer-readable medium

Abstract

Embodiments of the present invention provide an image processing method and apparatus, an electronic device, and a computer-readable medium. The method comprises: first performing feature extraction processing on an image to be processed, so as to obtain a feature extraction result; then, encoding the feature extraction result to obtain image feature representation data and detection query representation data; and finally, on the basis of the image feature representation data, the detection query representation data, and tracking query representation data corresponding to said image, performing decoding to obtain target description information of said image, wherein the target description information can represent some targets in said image.

Description

Image processing method, device, electronic device, and computer-readable medium

[0001] This application claims priority to Chinese patent application No. 202310787598.2 filed on June 29, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] Embodiments of the present disclosure relate to an image processing method, apparatus, electronic device, and computer-readable medium. Background Art

[0003] Target tracking is used to analyze the motion state information of some targets from video data (for example, their position in each frame of video image, etc.).

[0004] Summary of the Invention

[0005] The present disclosure provides an image processing method, an apparatus, an electronic device, and a computer-readable medium.

[0006] In order to achieve the above objectives, the technical solutions provided by the present disclosure are as follows:

[0007] The present disclosure provides an image processing method, the method comprising:

[0008] Perform feature extraction on the image to be processed to obtain a feature extraction result;

[0009] Encoding the feature extraction results to obtain detection query representation data;

[0010] Decoding is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0011] In one possible implementation, encoding the feature extraction result to obtain detection query representation data includes:

[0012] Encoding the feature extraction results to obtain image feature representation data and detection query representation data;

[0013] The decoding process is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed, including:

[0014] Decoding is performed based on the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed.

[0015] In one possible implementation, the plurality of historical images include first-category images and second-category images;

[0016] The process of determining the tracking query characterization data includes:

[0017] Extracting first tracking target description information from the target description information of the first type of image;

[0018] The tracking query representation data is determined according to the first tracking target description information and the target description information of the second type of image.

[0019] In a possible implementation manner, determining the tracking query representation data based on the first tracking target description information and the target description information of the second type of image includes:

[0020] determining second tracking target description information based on the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image;

[0021] The tracking query representation data is determined according to the second tracking target description information and the first tracking target description information.

[0022] In a possible implementation manner, the image to be processed and a plurality of historical images corresponding to the image to be processed belong to the same video data;

[0023] The time sequence of the image to be processed is later than the time sequence of the first type of images;

[0024] The timing of the first type of images is later than the timing of the second type of images.

[0025] In one possible implementation, decoding the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed includes:

[0026] Determining query representation data to be processed based on the detection query representation data and the tracking query representation data;

[0027] Decoding is performed based on the query representation data to be processed to obtain target description information of the image to be processed.

[0028] In one possible implementation, the decoding process is performed based on the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed, including:

[0029] Determining query representation data to be processed based on the detection query representation data and the tracking query representation data;

[0030] Decoding is performed based on the image feature representation data and the query representation data to be processed to obtain target description information of the image to be processed.

[0031] In one possible implementation, the object description information of the image to be processed is determined using a pre-trained decoder;

[0032] The decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between query representation data corresponding to different historical images belonging to the same target.

[0033] In one possible implementation, the at least one target includes one or more targets to be tracked;

[0034] The at least one decoding layer includes a first decoding layer and a second decoding layer;

[0035] The decoder also includes a feature fusion module;

[0036] The feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; the tracking query processing data is determined based on the first decoding layer and the tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each target to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each processed query representation data;

[0037] The second decoding layer is used to process the tracking query fusion data and the detection query processing data output by the first decoding layer; the detection query processing data is determined according to the first decoding layer and the detection query representation data.

[0038] In one possible implementation, the feature fusion module includes an information removal branch network and an information addition branch network;

[0039] The process of determining the tracking query fusion data includes:

[0040] Processing the tracking query processing data using the information removal branch network to obtain retained information features;

[0041] Processing the tracking query processing data using the information adding branch network to obtain features of the information to be added;

[0042] The tracking query fusion data is determined according to the retained information features and the to-be-added information features.

[0043] In one possible implementation, the process of determining the retained information feature includes:

[0044] Performing time series information extraction processing on the tracking query processing data to obtain first time series information;

[0045] Performing self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result;

[0046] Performing full connection processing on the self-attention processing result to obtain a full connection processing result;

[0047] Determining information representation data to be removed based on the full connection processing result;

[0048] The tracking query processing data is subjected to information removal processing according to the information representation data to be removed, so as to obtain the retained information features.

[0049] In one possible implementation, the process of determining the information feature to be added includes:

[0050] Performing time series information extraction processing on the tracking query processing data to obtain second time series information;

[0051] Self-attention processing is performed on the second time series information and the tracking query processing data to obtain the information features to be added.

[0052] In one possible implementation, the self-attention layer in the feature fusion module is implemented using a second attention mask, which is used to shield the interaction between query representation data belonging to different targets.

[0053] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0054] The process of determining the training loss of the target tracking model includes:

[0055] determining target description information of the first image data based on the first image data, target description information of a plurality of historical images corresponding to the first image data, and the target tracking model;

[0056] extracting a first set of information and a second set of information from the target description information of the first image data;

[0057] Obtaining a first loss based on the first set of information and target label data corresponding to the first image data;

[0058] Obtaining a second loss based on the second set of information and the target label data;

[0059] Determine a training loss of the target tracking model based on the first loss and the second loss.

[0060] In a possible implementation manner, the timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

[0061] In one possible implementation manner, obtaining the second loss based on the second set of information and the target label data includes:

[0062] The second loss is determined based on a label matching result, the second set of information, and the target label data; the label matching result is determined based on a matching result between the first set of information and the target label data.

[0063] In one possible implementation, the second set of information includes at least one target prediction result;

[0064] The process of determining the second loss includes:

[0065] Determining a loss corresponding to each target prediction result based on the label matching result, the second set of information, and the target label data;

[0066] The second loss is determined according to an average value of the losses corresponding to the at least one target prediction result.

[0067] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0068] The training process of the target tracking model includes:

[0069] Using at least one second image data and the target detection label of each second image data, training the initial model to obtain a model to be optimized;

[0070] The decoding module in the model to be optimized is trained using at least one image sequence and the target tracking label of each image sequence to obtain the target tracking model.

[0071] In a possible implementation manner, the at least one target includes one or more targets to be tracked; and the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

[0072] In a possible implementation manner, the image to be processed is an image extracted from an image sequence; the target description information includes at least one target prediction result;

[0073] After obtaining the target description information of the image to be processed, the method further includes:

[0074] Extracting target prediction results that meet preset reference conditions from the target description information;

[0075] The image to be processed is updated using the next frame image in the image sequence, the tracking query representation data is updated using the query representation data in the target prediction result that meets the preset reference condition, and the step of performing feature extraction processing on the image to be processed is continued.

[0076] In a possible implementation manner, the image sequence is video data.

[0077] The present disclosure provides an image processing device, comprising:

[0078] An extraction unit is configured to perform feature extraction processing on the image to be processed to obtain a feature extraction result;

[0079] an encoding unit configured to encode the feature extraction result to obtain detection query representation data;

[0080] A decoding unit is configured to perform decoding processing based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0081] The present disclosure provides an electronic device, the device comprising: a processor and a memory;

[0082] The memory is configured to store instructions or computer programs;

[0083] The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the image processing method provided by the present disclosure.

[0084] The present disclosure provides a computer-readable medium having instructions or a computer program stored therein. When the instructions or the computer program are executed on a device, the device executes the image processing method provided by the present disclosure.

[0085] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, wherein the computer program contains program codes for executing the image processing method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0087] FIG1 is a flow chart of an image processing method provided by an embodiment of the present disclosure;

[0088] FIG2 is a schematic diagram of a target tracking model provided by an embodiment of the present disclosure;

[0089] FIG3 is a schematic diagram of a feature fusion module provided in an embodiment of the present disclosure;

[0090] FIG4 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure;

[0091] FIG5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0092] Research has found that the two-stage tracking scheme based on the Kalman filter described above, because it relies on the Kalman filter's motion modeling, requires that the target's speed and direction cannot change too quickly. However, because targets in low-frame-rate videos change more dramatically between frames (for example, they may experience rapid changes in speed, direction, and appearance), this scheme's target tracking performance is severely degraded in low-frame-rate videos, making it unsuitable for processing low-frame-rate videos.

[0093] Based on the above research, in order to better improve the target tracking effect, the present disclosure provides an image processing method, which includes: for an image to be processed (for example, a frame of video image in a video data), first performing feature extraction processing on the image to be processed to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; then encoding the feature extraction result to obtain detection query representation data (for example, multiple detection queries); finally, decoding processing is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed (for example, a tracking query determined based on multiple historical images) to obtain target description information of the image to be processed (for example, target content feature representation data, target position representation data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, since the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed, the tracking query representation data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query representation data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0094] In addition, when the image processing method provided by the present invention is used to implement target tracking processing for video data, the image processing method needs to refer to the target description information of multiple historical images (for example, Query) to determine the target in a certain video image, so that in the process of determining the target of the video image, more reference content (for example, the state of the target in different historical images, etc.) can be obtained from the target description information of these historical images. This can effectively overcome the problems caused by the low frame rate of the video data, thereby effectively improving the target tracking effect for low frame rate videos, and thus making the image processing method provided by the present invention suitable for target tracking processing for video data with any frame rate, which is conducive to improving the universality of the image processing method in the field of target tracking.

[0095] In addition, the present disclosure does not limit the execution subject of the image processing method. For example, any implementation of the image processing method provided in the embodiments of the present disclosure can be applied to a device with data processing functions such as a terminal device or a server. For another example, any implementation of the image processing method provided in the embodiments of the present disclosure can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be a stand-alone server, a cluster server or a cloud server.

[0096] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0097] To better understand the technical solutions provided by the present disclosure, the image processing method provided by the present disclosure is described below with reference to some accompanying drawings. As shown in FIG1 , the image processing method provided by an embodiment of the present disclosure includes steps S1-S3 below. FIG1 is a flowchart of an image processing method provided by an embodiment of the present disclosure.

[0098] S1: Perform feature extraction on the image to be processed to obtain feature extraction results.

[0099] The image to be processed refers to the image data that needs to be processed for target determination (for example, the t-th frame video image I shown in FIG2 t ). t is a positive integer. It should be noted that the present disclosure does not limit the implementation of the target. For example, the target can refer to any detectable object (such as a person, an animal, a plant, an object, etc.).

[0100] In addition, the present disclosure is not limited to the above-mentioned image to be processed. For example, in some application scenarios (such as target tracking processing for a certain image sequence), the image to be processed refers to an image extracted from the image sequence (for example, the image data at the tth arrangement position). The image sequence is used to record multiple image data arranged in a certain order (for example, time sequence); and the present disclosure is not limited to the implementation method of the image sequence. For example, it can be implemented using any existing or future image sequence (for example, video data).

[0101] The feature extraction result is used to represent the image information carried by the above image to be processed.

[0102] In addition, the present disclosure does not limit the process of determining the above feature extraction results (that is, the implementation method of S1). For example, it can be implemented using any existing or future method that can perform feature extraction processing on image data (for example, with the help of a feature extractor, etc.).

[0103] It can be seen that in some application scenarios, the above S1 can specifically be: inputting the image to be processed into a pre-built feature extractor to obtain the feature extraction result output by the feature extractor, so that the feature extraction result can represent the image information carried by the image to be processed. Among them, the feature extractor is used to perform feature extraction processing on the input data of the feature extractor; and the present disclosure does not limit the implementation method of the feature extractor. For example, the feature extractor can be implemented using any existing or future device with a feature extraction function. For another example, the feature extractor can be implemented using a convolutional neural network (CNN). For another example, the feature extractor can be implemented using the feature extraction module in the target tracking model below.

[0104] It should be noted that the present disclosure does not limit the implementation method of CNN in the above paragraph. For example, it can be implemented using ResNet50 pre-trained based on the CoCo dataset.

[0105] Based on the above two paragraphs, it can be seen that in one possible implementation, when the image processing method provided by the present disclosure is implemented with the help of a pre-trained target tracking model (for example, the multi-target tracking module shown in Figure 2), the above S1 can specifically be: using the feature extraction module in the target tracking model (for example, the CNN shown in Figure 2) to perform feature extraction processing on the above image to be processed, and obtain the feature extraction result corresponding to the image to be processed.

[0106] Based on the relevant content of S1 above, it can be known that for the image to be processed (for example, the t-th frame video image I shown in FIG2 t ), after obtaining the image to be processed, feature extraction processing is performed on the image to be processed to obtain a feature extraction result (for example, the data output by the CNN shown in Figure 2), so that the feature extraction result can represent the image information carried by the image to be processed, so that some targets in the image to be processed can be determined based on the feature extraction result.

[0107] S2: Encode the feature extraction results to obtain detection query representation data.

[0108] Wherein, the detection query representation data is used to represent some targets in the above-mentioned image to be processed; and the present disclosure does not limit the detection query representation data. For example, the detection query representation data may include query representation data (e.g., Query) corresponding to multiple targets in the image to be processed. Wherein, the query representation data corresponding to the target is used to describe a target in the image to be processed. It can be seen that under a possible implementation method, the detection query representation data may include query representation data corresponding to one or more targets under the image to be processed. Wherein, the query representation data corresponding to the target under the image to be processed is used to represent the state of the target under the image to be processed.

[0109] In addition, the present disclosure does not limit the implementation method of the query representation data corresponding to the target in the above paragraph. For example, it may include target content feature representation data corresponding to the target and target position representation data corresponding to the target. The target content feature representation data is used to represent the features presented by the target in the image to be processed (for example, features presented in terms of posture, clothing color, etc.); and the present disclosure does not limit the implementation method of the target content feature representation data. For example, the target content feature representation data can be implemented using a feature vector. The target position representation data is used to represent the position of the target in the image to be processed; and the present disclosure does not limit the implementation method of the target content feature representation data. For example, it can be implemented using a bounding box.

[0110] In addition, the present disclosure does not limit the implementation method of the encoding process in S2 above. For example, it can be implemented using any existing or future method that can perform encoding processing (for example, with the help of an encoder, etc.).

[0111] It can be seen that, in one possible implementation, S2 above can specifically be: inputting the above feature extraction results into a pre-built encoder to obtain the detection query representation data output by the encoder. The encoder is used to encode the input data of the encoder; and the present disclosure does not limit the implementation of the encoder. For example, the encoder can be implemented using any existing or future device with encoding function. For another example, the encoder can be implemented using a Transformer Encoder. For another example, the encoder can be implemented using the encoding module in the target tracking model below.

[0112] It should be noted that the present disclosure does not limit the implementation of the Transformer Encoder in the above paragraph. For example, it can be implemented using the Transformer Encoder module in the DINO (DETR with Improved DeNoising Anchor Boxes) detection algorithm.

[0113] Based on the above two paragraphs, it can be seen that in one possible implementation, when the image processing method provided by the present disclosure is implemented with the help of a pre-trained target tracking model (for example, the multi-target tracking module shown in FIG2 ), the above S2 can be specifically as follows: after determining the feature extraction result from the above image to be processed by the feature extraction module in the target tracking model, the encoding module in the target tracking model performs encoding processing on the feature extraction result to obtain the detection query representation data corresponding to the image to be processed (for example, D shown in FIG2 ). t ).

[0114] It should be noted that for D shown in Figure 2 t For example, D t For representing the image data I shown in FIG. t some targets; and the D t It can be expressed using the following formula (1).

[0115] Where D t Indicates that by targeting the above image data I t Perform processing (e.g., feature extraction + encoding) on ​​the query; Indicates that the image data I t The query corresponding to the first target in Including target content feature representation data corresponding to the first target and target position representation data corresponding to the first target; Indicates that the image data I t The query corresponding to the second target in Including target content feature representation data corresponding to the second target and target position representation data corresponding to the second target; ... (and so on); Indicates that the image data I t Nth det The query corresponding to the target, and the Including the Nth det The target content feature representation data corresponding to the Nth target and the det The target position representation data corresponding to the target; N det Indicates the number of pre-set queries so that N det Used to indicate that D t The number of queries involved.

[0116] In fact, in some application scenarios, in order to better improve the target tracking effect, the present disclosure also provides a possible implementation method of the encoding process in S2 above. In this implementation method, S2 can specifically be: encoding the feature extraction results to obtain image feature representation data and detection query representation data. The image feature representation data is used to represent the information carried by the image to be processed above; and the present disclosure does not limit the implementation method of the image feature representation data. For example, it can be implemented using any existing or future feature encoding vector determined for image data (for example, the image feature map shown in Figure 2).

[0117] In addition, the present disclosure does not limit the implementation method of the encoding process in the above paragraph. For example, the encoding process may specifically be: inputting the above feature extraction result into a pre-built encoder to obtain image feature representation data and detection query representation data output by the encoder. For another example, when the image processing method provided by the present disclosure is implemented with the aid of a pre-trained target tracking model (for example, the multi-target tracking module shown in FIG2 ), the encoding process may specifically be: after determining the feature extraction result from the above image to be processed by the feature extraction module in the target tracking model, the encoding module in the target tracking model performs encoding processing on the feature extraction result to obtain image feature representation data corresponding to the image to be processed (for example, the image feature map shown in FIG2 ) and detection query representation data corresponding to the image to be processed (for example, the D image shown in FIG2 ). t ).

[0118] Based on the above content, it can be seen that in a possible implementation, for the image to be processed (for example, the t-th frame video image I shown in FIG2 ), t ), after obtaining the feature extraction result corresponding to the image to be processed (for example, the data output by the CNN shown in FIG2 ), the feature extraction result is encoded to obtain image feature representation data (for example, the image feature map shown in FIG2 ) and detection query representation data (for example, the D t ), so that some targets in the image to be processed can be determined based on the image feature representation data and the detection query representation data.

[0119] Based on the relevant content of S2 above, it can be known that for the image to be processed (for example, the t-th frame video image I shown in FIG2 t ), after obtaining the feature extraction result corresponding to the image to be processed (for example, the data output by the CNN shown in FIG2 ), the feature extraction result is encoded to obtain at least the detection query representation data (for example, the D shown in FIG2 ). t ), so that some targets in the image to be processed can be determined based on the detection query characterization data.

[0120] S3: Decoding is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0121] Among them, the tracking query representation data corresponding to the image to be processed refers to the tracking query required to be used as the basis for target determination processing on the image to be processed, so that the tracking query representation data can represent the status of some tracked targets in the image to be processed (for example, the "target to be tracked" below) in the historical stage.

[0122] In addition, the tracking query representation data corresponding to the image to be processed is determined based on the target description information of multiple historical images corresponding to the image to be processed. The time sequence of the historical images is earlier than the time sequence of the image to be processed. Moreover, the present disclosure does not limit the number of historical images corresponding to the image to be processed. For example, when the image to be processed is the t-th frame video image I shown in FIG2 t When the multiple historical images corresponding to the image to be processed may include the t-1 frame video image I shown in FIG. t-1 , the t-2th frame video image I t-2 , the t-3 frame video image I t-3 The target description information of the historical image refers to the result obtained when performing target determination processing on the historical image, so that the target description information of the historical image can describe some targets in the historical image.

[0123] It should be noted that the present disclosure does not limit the implementation of the target description information. For example, the target description information may include query representation data corresponding to some targets (e.g., the feature vector corresponding to the target + the bounding box corresponding to the target). For another example, the target description information may also include the category confidence corresponding to these targets. The category confidence is used to indicate the probability that the target belongs to a certain category.

[0124] It should also be noted that the present disclosure does not limit the implementation method of the “multiple” in the above “multiple historical images corresponding to the image to be processed”. For example, it can be 2, 3 or other positive integers greater than 1.

[0125] In addition, the present disclosure does not limit the implementation method of the above-mentioned process of determining the tracking query representation data corresponding to the image to be processed. For example, when the multiple historical images corresponding to the image to be processed include R image data, the process of determining the tracking query representation data may specifically include: determining the query representation data (Query) corresponding to some or all targets in the target description information of the r-th image data as the tracking Query corresponding to the r-th image data, where r is a positive integer, r≤R, and R is a positive integer; and then aggregating the tracking Queries corresponding to the R image data to obtain the tracking query representation data, so that the tracking query representation data includes the tracking Queries corresponding to some tracked targets in the multiple historical images corresponding to the image to be processed.

[0126] In fact, in order to better improve the target determination effect, the present disclosure also provides a possible implementation method of the process of determining the tracking query representation data corresponding to the image to be processed. In this implementation method, when the multiple historical images corresponding to the image to be processed include the first type of image (for example, the t-1 frame video image I shown in FIG2 t-1 ) and the second type of image (for example, the t-2 frame video image I shown in FIG. t-2 , the t-3 frame video image I t-3 When the tracking query representation data corresponding to the image to be processed is determined, the process may include the following steps 11 and 12.

[0127] Step 11: Extract first tracking target description information from the target description information of the first category of images.

[0128] Among them, the first type of image is used to provide the most recent historical position of the target tracked in the above image to be processed; and the present disclosure is not limited to the first type of image. For example, when the image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data, the timing of the image to be processed is later than the timing of the first type of image, and the timing of the first type of image is later than the timing of other images in the multiple historical images except the first type of image (that is, the second type of image above).

[0129] It can be seen that in a possible implementation, when the above-mentioned image to be processed and the multiple historical images corresponding to the image to be processed belong to the same video data, the above-mentioned first type of image refers to the historical image that exists in the multiple historical images and whose time sequence is closest to the time sequence of the image to be processed (for example, the t-1 frame video image I shown in FIG2 t-1 ).

[0130] The first tracking target description information is used to describe the most recent historical information extracted from the first type of image above, which is required to be referenced when determining some targets in the image to be processed above, so that the first tracking target description information can represent some tracked targets that appear in the historical image whose time sequence is closest to that of the image to be processed.

[0131] In addition, the present disclosure does not limit the implementation method of the above-mentioned first tracking target description information. For example, it may include target content feature representation data and target position representation data of at least one tracked target presented in the above-mentioned first type of image, so that the target content feature representation data can represent the state of the tracked target in the historical image whose time sequence is closest to the time sequence of the image to be processed, and the target position representation data can represent the position of the tracked target in the historical image whose time sequence is closest to the time sequence of the image to be processed.

[0132] In addition, the present disclosure does not limit the determination process of the first tracking target description information mentioned above. For example, it can be specifically as follows: after obtaining the target description information of the first type of image mentioned above, first extract at least one query that meets the preset filtering conditions from the target description information; then determine the first tracking target description information based on these queries, so that the first tracking target description information includes these queries. The preset filtering conditions can be pre-set. For example, the preset filtering conditions can be: the query is assigned a tracking object identifier (for example, ID), and the IOU (Intersection over Union) of the query is greater than a preset threshold. The tracking object identifier is used to uniquely identify the tracking process for a tracked target.

[0133] Based on the relevant content of step 11 above, it can be known that for the above image to be processed (for example, the t-th frame video image I shown in FIG2 t ), after obtaining a historical image (e.g., the t-1 frame video image I shown in FIG2 ) that is closest to the image to be processed and corresponds to the image to be processed, t-1 ) target description information, some target content feature representation data and target position representation data of the tracked targets can be extracted from the target description information to determine the first tracking target description information (for example, X shown in FIG. t The tracking query corresponding to t-1 in the image is used to enable the first tracking target description information to represent the status of the tracked targets in the historical image.

[0134] Step 12: Determine the tracking query representation data corresponding to the image to be processed based on the first tracking target description information and the target description information of the second type of image.

[0135] Among them, the second type of image is used to represent other images in the multiple historical images corresponding to the above-mentioned image to be processed, except for the above-mentioned first type of image; and the present disclosure does not limit the second type of image. For example, when the image to be processed and the multiple historical images corresponding to the image to be processed belong to the same video data, the timing of the image to be processed is later than the timing of the first type of image, and the timing of the first type of image is later than the timing of the second type of image.

[0136] It can be seen that in a possible implementation manner, when the above-mentioned image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data, the above-mentioned second type of image refers to the historical image whose time sequence among the multiple historical images is not the one closest to the time sequence of the image to be processed (for example, the t-2 frame video image I shown in FIG2 t-2 , the t-3 frame video image I t-3 wait).

[0137] In addition, the present disclosure does not limit the implementation method of the above step 12. For example, it can be specifically as follows: first extract some target content feature representation data of the tracked targets from the target description information of the second category of images above; then perform preset processing (for example, integration processing, fusion processing or splicing processing, etc.) on the target content feature representation data and the target content feature representation data of the corresponding targets in the above first tracking target description information to obtain the tracking query representation data corresponding to the above image to be processed, so that the tracking query representation data can not only represent the positions of these tracked targets in the most recent frame of historical images, but also can more comprehensively represent the features presented by these tracked targets in the most recent multiple frames of historical images.

[0138] In fact, in order to better improve the target determination effect, the present disclosure also provides another possible implementation of the above step 12, which may specifically include the following steps 121 and 122.

[0139] Step 121: Determine second tracking target description information based on the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image.

[0140] The second tracking target description information is used to describe the tracking query determined based on the second type of image and required to be referenced when determining some targets in the image to be processed.

[0141] In addition, the present disclosure does not limit the implementation of the above step 121. For example, it can be specifically as follows: combining the target position representation data in the above first tracking target description information and the target content feature representation data of the corresponding target in the above second type of image target description information to obtain at least one combined query of the tracked target, so that the combined query can represent the features presented by the tracked target in the second type of image and the position of the tracked target in the above first type of image; and then determining the second tracking target description information (for example, X shown in FIG. 2 ) based on these combined queries. t The tracking query corresponding to t-2 or the tracking query corresponding to t-3 in the tracking target description information is used to make the second tracking target description information include these combined queries.

[0142] Based on the relevant content of step 121 above, it can be known that for the above image to be processed (for example, the t-th frame video image I shown in FIG. 2 ), t ), when a historical image corresponding to the image to be processed is obtained and is not the closest to the image to be processed (for example, the video image frame t-2 shown in FIG. 2 ), t-2 , or the t-3 frame video image I t-3 etc.), after obtaining the target description information of the tracked target, some target content feature representation data of the tracked target can be extracted from the target description information, so that the target content feature representation data can represent the features presented by the tracked target in the historical image; then the target content feature representation data is combined with the target position representation data of the corresponding target in the first tracking target description information above to obtain the combined Query of the tracked target; finally, based on the combined Query, the second tracking target description information above is determined, so that the second tracking target description information can represent the tracking Query determined based on the historical image and required to be referenced when determining some targets in the image to be processed above.

[0143] Step 122: Determine tracking query representation data corresponding to the image to be processed according to the second tracking target description information and the first tracking target description information.

[0144] It should be noted that the present disclosure does not limit the implementation of the above step 122. For example, it can be specifically as follows: integrating the above second tracking target description information with the above first tracking target description information to obtain the tracking query representation data corresponding to the above image to be processed (for example, X shown in FIG2 t ), so that the tracking query representation data includes some tracking query sequences corresponding to the tracked targets (for example, the sequence shown in FIG. sequence and sequence wait).

[0145] It should also be noted that i, i+1, and i+2 shown in FIG2 respectively represent tracking object identifiers assigned to a tracked target, and i is a positive integer.

[0146] Based on the relevant contents of steps 121 to 122 above, it can be known that in some application scenarios, for the above image to be processed (for example, the t-th frame video image I shown in FIG. 2 ), t ), if the multiple historical images corresponding to the image to be processed include the first type of image (for example, the t-1 frame video image I shown in FIG2 t-1 ) and the second type of image (for example, the t-2 frame video image I shown in FIG. t-2 , the t-3 frame video image I t-3 etc.), the target position representation data in the target description information of the first type of image and the target content feature representation data in the target description information of the second type of image can be used to construct the tracking Query corresponding to the second type of image (that is, the second tracking target description information above); and then, combined with the tracking Query corresponding to the second type of image and the tracking Query corresponding to the first type of image (that is, the first tracking target description information above), the tracking query representation data corresponding to the above image to be processed is determined, so that the tracking query representation data can better represent the status of some tracked targets in the image to be processed in the historical stage.

[0147] Based on the relevant contents of steps 11 to 12 above, it can be known that in some application scenarios, for the above image to be processed (for example, the t-th frame video image I shown in FIG2 ), t ), we can use the N closest to the image to be processed in the historical time period max Frame history image (for example, the t-1 frame video image I shown in FIG2 t-1 , the t-2th frame video image I t-2 , the t-3 frame video image I t-3 etc.), and the target content feature representation data determined by the frame of historical image closest to the image to be processed (for example, the t-1 frame of video image I shown in FIG2 ). t-1 etc.), construct the tracking query representation data corresponding to the image to be processed (for example, the X t), so that all tracking queries recorded in the tracking query representation data can be used to collaboratively track some targets in the image to be processed, so that the tracking query representation data can be used to determine some tracked targets appearing in the image to be processed, and the above detection query representation data (for example, D shown in FIG2 ) can be used to determine some tracked targets appearing in the image to be processed. t ) is used to detect newly appearing targets in the image to be processed, so that target description information of the image to be processed can be determined based on the tracking query representation data and the detection query representation data.

[0148] Based on the relevant content of the tracking query representation data corresponding to the image to be processed above, it can be known that in one possible implementation, when there is at least one target in the image to be processed above, if the at least one target includes Y targets to be tracked, then the tracking query representation data includes the query representation data corresponding to the first target to be tracked under multiple historical images, the query representation data corresponding to the second target to be tracked under multiple historical images, ... (and so on), and the query representation data corresponding to the Yth target to be tracked under multiple historical images. Among them, the yth target to be tracked refers to a target that exists in the image to be processed and has been tracked in the historical stage; and the query representation data corresponding to the yth target to be tracked in a historical image is used to represent the state of the yth target to be tracked in the historical image, y is a positive integer, y≤Y, and Y is a positive integer.

[0149] Based on the content of the previous paragraph, it can be seen that for the tracking query representation data corresponding to the above-mentioned image to be processed, if the tracking query representation data is used to describe the historical status of one or more tracked targets, then the tracking query representation data includes at least two tracking queries for each tracked target, so that in the subsequent processing process (for example, the decoding process, etc.), for each tracked target, each tracking query corresponding to the target is used as an independent query to complete the tracking prediction processing for the target, and each query will make full use of the information carried in other queries corresponding to the target (for example, target content feature representation data + target position representation data, etc.) to complete its own iterative update processing to achieve collaborative tracking, so that the collaborative tracking process executed for each query is used to jointly track the same target, which can better improve the reliability of the features predicted for the target, so that the features predicted for the target can better express the state of the target in the image to be processed, which is conducive to improving the target tracking effect.

[0150] The above “target description information of the image to be processed” is used to describe at least one target in the image to be processed; and the present disclosure does not limit the implementation method of the “target description information of the image to be processed”. For example, the “target description information of the image to be processed” may include query representation data (Query) of some targets and the category confidence of these targets. Among them, the query representation data of the target may include target content feature representation data of the target and target position representation data of the target (such as a bounding box, etc.). The category confidence of the target is used to indicate how likely the target belongs to certain categories (such as people, animals, cars, etc.).

[0151] For another example, when the detection query representation data corresponding to the above image to be processed includes multiple detection queries, and the tracking query representation data corresponding to the above image to be processed includes multiple tracking queries, the target description information of the image to be processed may include the target prediction results corresponding to each detection query and the target prediction results corresponding to each tracking query. Among them, the target prediction result corresponding to the j-th detection query refers to the state of a target in the image to be processed derived based on the j-th detection query; j is a positive integer, j≤J, J is a positive integer, and J represents the number of the detection query. The target prediction result corresponding to the m-th tracking query refers to the state of a target in the image to be processed derived based on the m-th tracking query; m is a positive integer, m≤M, M is a positive integer, and M represents the number of the tracking query.

[0152] In addition, the present disclosure does not limit the process of determining the "target description information of the image to be processed" (i.e., the implementation of S3 above). For example, it can be implemented with the help of a pre-built decoder. The decoder is used to decode the input data of the decoder. Moreover, the present disclosure does not limit the decoder. For example, the decoder can be implemented using the decoding module in the multi-target tracking model shown in Figure 2.

[0153] Furthermore, in some application scenarios, when S2 above is implemented by "encoding the feature extraction results to obtain image feature representation data and detection query representation data," S3 above may specifically be: decoding the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain target description information for the image to be processed. It should be noted that the implementation of S3 shown in this paragraph may also be implemented using a pre-built decoder.

[0154] Furthermore, in order to better improve the target determination effect, the present disclosure also provides a possible implementation of the above S3, which may specifically include the following steps 21 and 22.

[0155] Step 21: Determine the query representation data to be processed based on the above detection query representation data and the above tracking query representation data.

[0156] Among them, the query representation data to be processed is used to describe all the queries (for example, multiple detection queries + multiple tracking queries) that need to be referenced when performing target determination processing on the above image to be processed, so that the query representation data to be processed can better represent the characteristics of some targets in the image to be processed.

[0157] In addition, the present disclosure does not limit the determination process of the above-mentioned query representation data to be processed. For example, it can be specifically: cascade processing (that is, splicing processing) of the above-mentioned detection query representation data and the above-mentioned tracking query representation data to obtain the query representation data to be processed, so that the query representation data to be processed includes the detection query representation data and the tracking query representation data, so that the query representation data to be processed can better represent the characteristics of some targets in the image to be processed.

[0158] Step 22: Decode the query representation data to be processed to obtain target description information of the image to be processed.

[0159] It should be noted that the present disclosure does not limit the implementation method of the above step 22. For example, it can be specifically: inputting the above query representation data to be processed into a pre-built decoder (for example, the decoding module in the multi-target tracking model shown in Figure 2), so that the decoder can perform decoding processing based on the query representation data to be processed, obtain and output the target description information of the above image to be processed (for example, the data output by the last decoding layer shown in Figure 2), so that the target description information can describe some targets in the image to be processed.

[0160] For example, in one possible implementation, when the above S2 is implemented by the step of "encoding the feature extraction results to obtain image feature representation data and detection query representation data", the step 22 can specifically be: decoding the image feature representation data and the above query representation data to be processed to obtain the target description information of the image to be processed.

[0161] It should be noted that the present disclosure does not limit the implementation method of the step in the previous paragraph of "decoding the image feature representation data and the query representation data to be processed to obtain the target description information of the image to be processed". For example, it can be specifically: inputting the image feature representation data and the query representation data to be processed into a pre-built decoder (for example, the decoding module in the multi-target tracking model shown in Figure 2), so that the decoder can decode the image feature representation data and the query representation data to be processed to obtain and output the target description information of the image to be processed (for example, the data output by the last decoding layer shown in Figure 2), so that the target description information can describe some targets in the image to be processed.

[0162] Based on the relevant content of steps 21 to 22 above, it can be seen that in some application scenarios, for the image to be processed, after obtaining the detection query representation data corresponding to the image to be processed and the tracking query representation data corresponding to the image to be processed, the two queries can be cascaded first; and then the cascade results can be directly sent to a pre-built decoder so that the decoder can output the target description information of the image to be processed.

[0163] Based on the above content, it can be seen that in one possible implementation, when the image processing method provided by the present disclosure is implemented with the help of a pre-trained target tracking model (for example, the multi-target tracking module shown in FIG2 ), the above S3 can be specifically as follows: for the above image to be processed (for example, the t-th frame video image I shown in FIG2 ), t ), after obtaining the tracking query representation data corresponding to the image to be processed (for example, X shown in FIG. t ), the detection query representation data corresponding to the image to be processed output by the encoding module in the target tracking model (for example, D shown in FIG. t ) and the image feature representation data corresponding to the image to be processed (for example, the image feature map shown in FIG2 ), the tracking query representation data and the detection query representation data can be cascaded first; and then the decoding module in the target tracking model performs decoding processing based on the cascade result and the image feature representation data, so that the decoding module can obtain and output the target description information of the image to be processed (for example, the data output by the last decoding layer shown in FIG2 ) in an autoregressive manner, so that the target description information includes some target queries and the category confidence of these targets.

[0164] Based on the above, it can be seen that in one possible implementation, the target description information of the image to be processed is determined using a pre-trained decoder. The decoder can be used to decode the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed; alternatively, the decoder can be used to decode the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed.

[0165] In addition, to further improve the target determination effect, the present disclosure also provides a possible implementation of the above decoder. Under this implementation, the decoder may include at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask. The first attention mask is used to shield the interaction between the query representation data corresponding to the same target in different historical images, so that the decoding layer achieves the effect of temporal blocking. It should be noted that the present disclosure does not limit the number of decoding layers; for example, it can be 6.

[0166] It should be noted that the present disclosure does not limit the implementation method of the "query representation data corresponding to the same target in different historical images" in the previous paragraph. For example, when the decoder shown in the previous paragraph refers to the h-th decoder, the "query representation data corresponding to the same target in different historical images" refers to the different tracking queries corresponding to the same target existing in the input data of the self-attention module in the h-th decoder; h is a positive integer, h≤the number of decoders (for example, 6).

[0167] Based on the content of the previous paragraph, it can be seen that in one possible implementation, the above decoder can be composed of a stack of multiple (for example, 6) temporal blocking decoder layers, and the self-attention modules in each temporal blocking decoder layer are implemented using a first attention mask, so that each self-attention module can use the first attention mask to shield the interaction between the query representation data corresponding to the same target in different historical images, so as to avoid mutual inhibition between different tracking queries of the same target.

[0168] The first attention mask mentioned above refers to the attention mask configured for the self-attention module in the decoding layer above, so that the self-attention module can use the attention mask to shield the interaction between the query representation data corresponding to the same target in different historical images; and the first attention mask is configured to shield the interaction between the query representation data corresponding to the same target in different historical images. In addition, the present disclosure does not limit the implementation of the first attention mask. For example, the first attention mask in the self-attention module in the decoding layer shown in Figure 2 can be implemented using the attention mask shown in Table 1 below.

[0169] Table 1 A possible implementation of the attention mask in the self-attention module in the decoding layer It should be noted that, for the above Table 1, Indicates that the target identified by i as the tracking object is in the image data I t-1 The corresponding tracking query; Indicates that the target identified by i+1 is in the image data I t-1 The corresponding tracking query; Indicates that the target identified by i+1 is in the image data I t-2 The corresponding tracking query; Indicates that the target identified by i+1 is in the image data I t-3 The corresponding tracking query is as follows; ... (and so on).

[0170] Based on the relevant content of the above decoder, it can be known that in one possible implementation, the above decoder can be composed of a stack of multiple temporally blocked decoding layers, and the attention mask in the self-attention module in each temporally blocked decoding layer can be configured to shield the interaction between different tracking queries belonging to the same target, so as to avoid mutual inhibition between different tracking queries for the same target. For example, for the self-attention module in the decoding layer, the self-attention module can be used not only for interaction processing between queries for different targets, but also for interaction processing between the same query; however, the self-attention module will shield the interaction processing between different tracking queries under the same target.

[0171] It can be seen that since the various input information of the self-attention module can interact equally in the traditional self-attention module, in order to effectively avoid the mutual suppression of multiple tracking queries for the same target, a mask (for example, the first attention mask mentioned above or the mask shown in Table 1 above) can be added to the self-attention module to obtain the self-attention module in the decoding layer above, so that the self-attention module in the decoding layer is the self-attention module with the mask added. Among them, because the interaction between different tracking queries belonging to the same target can be shielded in the self-attention module with the mask added, this can effectively avoid the phenomenon of multiple tracking queries for the same target suppressing each other in the self-attention module with the mask added, thereby improving the target tracking effect.

[0172] In fact, in order to better improve the decoding effect, the present disclosure also provides another possible implementation of the above decoder. In this implementation, when the decoder includes at least one decoding layer, and the at least one decoding layer includes a first decoding layer and a second decoding layer, the decoder may further include a feature fusion module (for example, the feature fusion module shown in Figure 2). The first decoding layer is used to represent a decoding layer that is closer to the front of any two adjacent decoding layers in the at least one decoding layer. The second decoding layer is used to represent a decoding layer that is closer to the back of any two adjacent decoding layers in the at least one decoding layer. The feature fusion module is located between the first decoding layer and the second decoding layer. For ease of understanding, the following is explained with reference to examples.

[0173] As an example, when the above decoder includes 6 decoding layers arranged in sequence, if the first decoding layer is the 1st decoding layer in the decoder, the second decoding layer is the 2nd decoding layer in the decoder, and there is a feature fusion module between the 1st decoding layer and the 2nd decoding layer; if the first decoding layer is the 2nd decoding layer in the decoder, the second decoding layer is the 3rd decoding layer in the decoder, and there is a feature fusion module between the 2nd decoding layer and the 3rd decoding layer; ... (and so on).

[0174] Based on the above two paragraphs, it can be seen that for any triplet (first decoding layer, feature fusion module, second decoding layer) present in the above decoder, the association relationship between the different elements in the triplet can be specifically as follows: the feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; and the second decoding layer is used to process the tracking query fusion data output by the feature fusion module and the detection query processing data output by the first decoding layer. Among them, the tracking query processing data is determined based on the first decoding layer and the above tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each target to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each processed query representation data; the detection query processing data is determined based on the first decoding layer and the above detection query representation data.

[0175] The tracking query processing data mentioned above refers to the Query output by the first decoding layer above for the tracked target; and the tracking query processing data is determined based on the first decoding layer and the tracking query representation data corresponding to the image to be processed above. For example, if the first decoding layer is the first decoding layer in the decoder mentioned above, the tracking query processing data output by the first decoding layer refers to the Query obtained by processing the tracking query representation data mentioned above by the first decoding layer, so that the tracking query processing data output by the first decoding layer includes the processed query representation data output by the first decoding layer for each tracked target; if the first decoding layer is the second decoding layer in the decoder mentioned above, the tracking query processing data output by the first decoding layer refers to the Query obtained by processing the Query for describing the tracked target present in the input data of the second decoding layer by the second decoding layer. to the Query, so that the tracking query processing data output by the first decoding layer includes the processed query representation data output by the second decoding layer for each tracked target; if the first decoding layer is the third decoding layer in the decoder above, then the tracking query processing data output by the first decoding layer refers to the Query obtained by the third decoding layer processing the Query for describing the tracked target present in the input data of the third decoding layer, so that the tracking query processing data output by the first decoding layer includes the processed query representation data output by the third decoding layer for each tracked target; ... (and so on). Wherein, the processed query representation data refers to the Query obtained by the first decoding layer iteratively for a tracked target.

[0176] The detection query processing data mentioned above refers to the query output by the first decoding layer above for the target to be detected; and the detection query processing data is determined based on the first decoding layer and the detection query representation data mentioned above. For example, if the first decoding layer is the first decoding layer in the decoder mentioned above, the detection query processing data output by the first decoding layer refers to the query obtained by the first decoding layer processing the detection query representation data mentioned above; if the first decoding layer is the second decoding layer in the decoder mentioned above, the detection query processing data output by the first decoding layer refers to the query obtained by the second decoding layer processing the query for describing the target to be detected in the input data of the second decoding layer; if the first decoding layer is the third decoding layer in the decoder mentioned above, the detection query processing data output by the first decoding layer refers to the query obtained by the third decoding layer processing the query for describing the target to be detected in the input data of the third decoding layer; ... (and so on).

[0177] The tracking query fusion data mentioned above refers to the fusion result obtained by the feature fusion module above for the query associated with each tracked target output by the first decoding layer above, so that the tracking query fusion data includes the fused query representation data output by the feature fusion module for each tracked target, so that the tracking query fusion data can better represent the characteristics presented by each tracked target. Among them, the fused query representation data refers to the fusion result obtained by fusing the tracked target and the query associated with the tracked target. For example, for the tracked target A, if the target A corresponds to 3 tracking query processing data Query A-1 , Query A-2 , Query A-3 , then in this Query A-1 In the corresponding fusion process, the Query A-1 With Query A-2 , Query A-3 The information carried is interacted to obtain the Query A-1 The corresponding fused query representation data Query A-1 ';In the Query A-2 In the corresponding fusion process, the Query A-2 With Query A-1 , Query A- 3 to interact with the information carried by the query A-2The corresponding fused query representation data Query A-2 ';In the Query A-3 In the corresponding fusion process, the Query A-3 With Query A-1 , Query A-2 The information carried is interacted to obtain the Query A-3 The corresponding fused query representation data Query A-3 '.

[0178] Furthermore, this disclosure does not limit the implementation of the feature fusion module described above. For example, in some application scenarios, the feature fusion module may include multiple fusion sub-modules, each of which is used to perform fusion processing on tracking queries associated with different tracked targets. Therefore, the number of fusion sub-modules can be determined based on the number of tracked targets. For example, in another possible implementation, the feature fusion module can be used to perform the following steps: fusing the tracking queries associated with the first tracked target to obtain and output fused query representation data corresponding to the first tracked target, so that the fused query representation data corresponding to the first tracked target includes the fusion results corresponding to each tracking query associated with the first tracked target, thereby making the number of queries in the fused query representation data corresponding to the first tracked target consistent with the number of tracking queries associated with the first tracked target; fusing the tracking queries associated with the second tracked target to obtain and output the fused query representation data corresponding to the second tracked target, so that the fused query representation data corresponding to the second tracked target includes the fusion results corresponding to each tracking query associated with the second tracked target, thereby making the number of queries in the fused query representation data corresponding to the second tracked target consistent with the number of tracking queries associated with the second tracked target; ... (and so on).

[0179] In fact, in order to better improve the fusion efficiency, the present disclosure also provides a possible implementation method of the above-mentioned feature fusion module. Under this implementation method, the feature fusion module includes an information removal branch network and an information addition branch network; and the process of using the feature fusion module to determine the above-mentioned tracking query fusion data can specifically include the following steps 31-33.

[0180] Step 31: Use the above information removal branch network to process the above tracking query processing data to obtain the retained information features.

[0181] Among them, the information removal branch network is used to perform information removal processing on the input data of the above feature fusion module; and the present disclosure does not limit the information removal branch network. For example, the information removal branch network can include two self-attention layers, a fully connected layer and a sigmoid layer.

[0182] The retained information feature is used to represent the information retained after the information removal process is performed on the input data of the feature fusion module above.

[0183] In addition, the present disclosure does not limit the determination process of the above retained information features. For example, it may specifically include the following steps 311 to 315.

[0184] Step 311: performing timing information extraction processing on the above tracking query processing data to obtain first timing information, so that the first timing information can represent the timing information carried in the tracking query processing data.

[0185] It should be noted that the present disclosure does not limit the implementation of step 311. For example, step 311 can be implemented with the help of a self-attention layer.

[0186] Based on the relevant content of step 311 above, it can be known that for the above information removal branch network (for example, the information removal branch network shown in FIG3 ), the information removal branch network receives the tracking query processing data output by the above first decoding layer (for example, the information removal branch network shown in FIG3 ). ), the first self-attention layer in the information removal branch network can perform timing information extraction processing on the tracking query processing data to obtain the first timing information (for example, the timing information 1 shown in Figure 3), so that the first timing information can represent the timing information carried in the tracking query processing data.

[0187] It should be noted that for the For example, the It can be represented by the first decoding layer in FIG2 for X shown in FIG2 t the results of the processing; and F t Used to describe the feature information predicted by the first decoding layer for some tracked targets; Represents the position information predicted by the first decoding layer for some tracked targets.

[0188] Step 312: Perform self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result.

[0189] It should be noted that the present disclosure does not limit the implementation of step 312. For example, step 312 can be implemented with the help of a self-attention layer.

[0190] Based on the relevant content of step 312 above, it can be known that for the above information removal branch network (for example, the information removal branch network shown in Figure 3), after the first self-attention layer in the information removal branch network outputs the above first timing information (for example, the timing information 1 shown in Figure 3), the second self-attention layer in the information removal branch network can perform self-attention processing based on the first timing information and the above tracking query processing data to obtain a self-attention processing result, so that the self-attention processing result can indicate which information needs to be eliminated.

[0191] Step 313: Perform full connection processing on the above self-attention processing result to obtain a fully connected processing result.

[0192] It should be noted that the present disclosure does not limit the implementation of step 313. For example, step 313 can be implemented with the help of a fully connected layer.

[0193] Based on the relevant content of step 313 above, it can be known that for the above information removal branch network (for example, the information removal branch network shown in Figure 3), after the second self-attention layer in the information removal branch network outputs the above self-attention processing result, the fully connected layer in the information removal branch network can perform fully connected processing on the self-attention processing result to obtain a fully connected processing result.

[0194] Step 314: Determine the information representation data to be removed based on the above full-connection processing result, so that the information representation data to be removed can indicate which information in the above tracking query processing data needs to be removed.

[0195] It should be noted that the present disclosure does not limit the implementation of step 314. For example, step 314 can be implemented with the help of a sigmoid layer.

[0196] Based on the relevant content of step 314 above, it can be known that for the above information removal branch network (for example, the information removal branch network shown in Figure 3), after the fully connected layer in the information removal branch network outputs the above fully connected processing result, the sigmoid layer in the information removal branch network can process the fully connected processing result to obtain the information representation data to be removed (for example, Z shown in Figure 3), so that the information representation data to be removed can indicate which information in the above tracking query processing data needs to be removed.

[0197] Step 315: Based on the above information representation data to be removed, perform information removal processing on the above tracking query processing data to obtain retained information features.

[0198] It should be noted that the present disclosure does not limit the implementation of step 315. For example, step 315 may specifically include: first, based on the above information representation data to be removed, determining the information representation data to be retained (for example, 1-Z shown in FIG3); then, multiplying the target content feature representation data in the above tracking query processing data by the information representation data to be retained to obtain the retained information feature (for example, F shown in FIG3). t ×(1-Z)).

[0199] Based on the relevant contents of steps 311 to 315 above, it can be known that, in one possible implementation, the above information removal branch network may include two self-attention layers, a fully connected layer and a sigmoid layer; wherein, the first self-attention layer is used to collect temporal information, the second self-attention layer is used to fuse the features of multiple queries, and the fully connected layer is used to perform fully connected processing on the output data of the second self-attention layer; the sigmoid layer is used to process the output data of the fully connected layer to obtain a gating value (that is, the information representation data to be removed above), so that the information removal processing can be performed on the input data of the information removal branch network based on the gating value (for example, Z shown in Figure 3).

[0200] Based on the relevant content of step 31 above, it can be known that for the above feature fusion module, if the feature fusion module includes an information removal branch network, then the tracking query processing data output by the first decoding layer above is received in the feature fusion module (for example, the tracking query processing data shown in FIG3 ) After that, the information removal branch network performs information removal processing on the tracking query processing data to obtain the retained information features (for example, F shown in FIG3 t ×(1-Z)), so as to subsequently determine the query fusion result for the tracking query processing data based on the retained information feature.

[0201] Step 32: Use the above information to add a branch network to process the above tracking query processing data to obtain the information features to be added.

[0202] Among them, the information adding branch network is used to extract the required added information from the input data of the above feature fusion module; and the present disclosure does not limit the information adding branch network. For example, the information adding branch network can include two self-attention layers.

[0203] The feature of the information to be added refers to the information required as a basis for the information addition process.

[0204] In addition, the present disclosure does not limit the process of determining the information features to be added above. For example, it may specifically include the following steps 321 and 322.

[0205] Step 321: Perform timing information extraction processing on the above tracking query processing data to obtain second timing information, so that the second timing information can represent the timing information carried in the tracking query processing data.

[0206] It should be noted that the present disclosure does not limit the implementation of step 321. For example, step 321 can be implemented with the help of a self-attention layer.

[0207] Based on the relevant content of step 321 above, it can be known that for the above information adding branch network (for example, the information adding branch network shown in FIG3 ), the information adding branch network receives the tracking query processing data output by the above first decoding layer (for example, the information adding branch network shown in FIG3 ). ), the first self-attention layer in the information-added branch network can perform timing information extraction processing on the tracking query processing data to obtain second timing information (for example, timing information 2 shown in FIG3 ), so that the second timing information can represent the timing information carried in the tracking query processing data.

[0208] Step 322: Perform self-attention processing on the second time series information and the tracking query processing data to obtain the information features to be added.

[0209] It should be noted that the present disclosure does not limit the implementation of step 322. For example, step 322 can be implemented with the help of a self-attention layer.

[0210] Based on the relevant content of step 322 above, it can be known that for the above information removal branch network (for example, the information addition branch network shown in FIG3 ), after the first self-attention layer in the information addition branch network outputs the above second time series information (for example, the time series information 2 shown in FIG3 ), the second self-attention layer in the information removal branch network can perform self-attention processing based on the second time series information and the above tracking query processing data to obtain the information feature to be added (for example, the time series information 2 shown in FIG3 ). ) so that the feature of the information to be added can represent the information required as a basis for information addition processing.

[0211] Based on the relevant content of steps 321 to 322 above, it can be seen that in one possible implementation, the above information adding branch network may include two self-attention layers; wherein the first self-attention layer is used to collect time series information, and the second self-attention layer is used to fuse the features of multiple queries.

[0212] Based on the relevant content of step 32 above, it can be known that for the above feature fusion module, if the feature fusion module includes an information adding branch network, then the tracking query processing data output by the first decoding layer above is received in the feature fusion module (for example, the tracking query processing data shown in FIG3 ) After that, the information adding branch network performs information extraction processing on the tracking query processing data to obtain the information features to be added (for example, the information features shown in FIG3 ), so as to subsequently determine the query fusion result for the tracking query processing data based on the information feature to be added.

[0213] Step 33: Determine the tracking query fusion data based on the retained information features and the to-be-added information features.

[0214] It should be noted that the present disclosure does not limit the implementation of the above step 33. For example, it can be implemented using the following formula (2).

[0215] Where, represents the output of the feature fusion module above (for example, the tracking query fusion data above); LN() represents the hierarchical normalization process; F t represents the target content feature representation data in the input data of the feature fusion module above; Z represents the gate value output by the sigmoid layer in the information removal branch network in the feature fusion module above; F t ×(1-Z) represents the data output by the information removal branch network in the feature fusion module above (for example, the information-retaining features above); Represents the data output by the second self-attention layer in the information addition branch network in the feature fusion module above (for example, the information features to be added above).

[0216] Based on the relevant contents of steps 31 to 33 above, it can be seen that for the above feature fusion module, the feature fusion module can include two branch networks, namely, the information removal branch network and the information addition branch network. Each branch network contains two superimposed self-attention layers, the first self-attention layer is used to collect time series information, and the second self-attention layer is used to fuse the features of multiple queries based on the time series information. In addition, for the information removal branch network, the output data of the second self-attention layer in the information removal branch network will further pass through a fully connected layer and a sigmoid layer to obtain a gated value. The gated value and the output data of the information addition branch network jointly determine the fused query information (as shown in formula (2) above). It can be seen that because the feature fusion module simulates the information interaction process through two branches, for a certain tracking query of any target, the feature fusion module can more efficiently integrate the valid information carried by other tracking queries of the target other than the certain tracking query into the certain tracking query to complete the self-iterative update for the certain tracking query, which is conducive to improving the interaction efficiency and effect between different tracking queries of the same target, thereby improving the target tracking effect.

[0217] In addition, for the above feature fusion module, in order to ensure that the feature fusion module can simultaneously perform query fusion processing for all tracked targets, the present disclosure also provides a possible implementation method of the feature fusion module. In this implementation method, the self-attention layer in the feature fusion module is implemented using a second attention mask, which is used to shield the interaction between query representation data belonging to different targets. This ensures that only interactions between all tracking queries of the same target are performed in the feature fusion module.

[0218] The second attention mask refers to the attention mask configured for the self-attention layer in the feature fusion module above; and the second attention mask is configured to block the interaction between query representation data belonging to different targets. In addition, this disclosure does not limit the implementation of the second attention mask. For example, the second attention mask in the self-attention layer in the feature fusion module shown in Figure 2 can be implemented using the attention mask shown in Table 2 below.

[0219] It should be noted that for each branch network in the above feature fusion module (for example, the information removal branch network or the information addition branch network), since the branch network includes two self-attention layers, in one possible implementation, the second attention mask used in each self-attention layer in the branch network can be implemented using an attention mask similar to that shown in Table 2 below.

[0220] It should also be noted that for the following Table 2, the as well as See above for more information.

[0221] Table 2 A possible implementation of the attention mask in the self-attention layer in the feature fusion module

[0222] Based on the relevant content of the feature fusion module above, it can be seen that the feature fusion module can be used to simultaneously perform query fusion processing for all tracked targets; and by configuring the attention mask in the self-attention layer in the feature fusion module to avoid interaction between queries of different targets, it can be ensured that only queries of the same target interact within the feature fusion module.

[0223] In addition, for each branch network in the above feature fusion module (for example, the information removal branch network or the information addition branch network), since the branch network includes two self-attention layers, the second attention mask used by the g-th self-attention layer in the branch network is used to shield the interaction between the query representation data of different targets appearing in the input data of the g-th self-attention layer, where g is a positive integer, g≤2.

[0224] Based on the relevant content of the decoder above, it can be seen that in one possible implementation, the decoder can be composed of a stack of multiple time-blocked decoding layers, and a feature fusion module will be inserted between any two adjacent decoding layers, so as to ensure that the decoder has a better decoding effect.

[0225] Based on the relevant contents of S1 to S3 above, it can be seen that for the image processing method provided by the embodiment of the present disclosure, feature extraction processing is first performed on the image to be processed to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; then the feature extraction result is encoded to obtain at least detection query representation data (e.g., multiple detection queries); finally, decoding processing is performed based on at least the detection query representation data and the tracking query representation data corresponding to the image to be processed (e.g., the tracking query determined based on multiple historical images) to obtain target description information of the image to be processed (e.g., target content feature representation data, target location representation data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed, the tracking query representation data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query representation data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0226] In fact, in some application scenarios, after obtaining the target description information of an image data, some high-quality target prediction results can be selected from the target description information as the target determination results of the image data. Based on this, the present disclosure also provides a possible implementation method of the above image processing method. Under this implementation method, when the target description information of the above image to be processed includes at least one target prediction result (for example, target prediction results corresponding to multiple detection queries + target prediction results corresponding to multiple tracking queries), the image processing method, in addition to including S1-S3 above, may also include the following step 41. The execution time of step 41 is later than the execution time of S3 above.

[0227] Step 41: After obtaining the target description information of the image to be processed, extract the target prediction result that meets the preset reference conditions from the target description information to obtain the target determination result of the image to be processed.

[0228] Among them, the preset reference condition refers to the condition required to screen high-quality target prediction results from the target description information of the image to be processed above; and the present disclosure does not limit the preset reference condition. For example, it can be: the target prediction result corresponding to the detection query or the target prediction result corresponding to the tracking query of the most recent historical image, and the category confidence reaches the preset threshold.

[0229] The target determination result of the image to be processed is used to describe some targets in the image to be processed that are predicted more accurately.

[0230] In addition, the present disclosure does not limit the determination process of the target determination result of the above-mentioned image to be processed. For example, it can be specifically as follows: after obtaining the target description information of the image to be processed, the target prediction result corresponding to the detection query and the target prediction result corresponding to the tracking query of the most recent historical image are extracted from the target description information, and both are used as the prediction results to be processed; then, it is determined whether the category confidence in each prediction result to be processed reaches a preset threshold. If so, it can be determined that the target described by the prediction result to be processed is relatively accurate. Therefore, the target determination result of the image to be processed can be determined based on the prediction result to be processed, so that the target determination result includes the prediction result to be processed.

[0231] Based on the relevant content of step 41 above, it can be seen that in some application scenarios, after determining the target description information of an image data, some high-quality target prediction results are extracted from the target description information to obtain the target determination result of the image data, so that the target determination result can better represent which targets exist in the image data, which is conducive to improving the target determination effect.

[0232] In fact, when the image processing method provided by the embodiment of the present disclosure is applied to the field of target tracking, after obtaining the target description information of a certain frame of image data, the target description information can be further used to determine the tracking query representation data corresponding to the next frame of image, which is conducive to improving the target determination effect for the next frame of image. Based on this, the present disclosure also provides a possible implementation method of the above image processing method. Under this implementation method, when the above image to be processed refers to an image extracted from an image sequence (for example, video data), and the target description information of the image to be processed includes at least one target prediction result (for example, target prediction results corresponding to multiple detection queries + target prediction results corresponding to multiple tracking queries), the image processing method can at least include S1-S3 above and steps 51-52 below. Among them, the execution time of step 51 is later than the execution time of S3.

[0233] Step 51: After obtaining the target description information of the image to be processed, extract the target prediction result that meets the preset reference conditions from the target description information.

[0234] It should be noted that the relevant content of step 51 is similar to part of the content of step 41 above, and for the sake of brevity, it will not be repeated here.

[0235] Step 52: Use the next frame image in the above image sequence to update the image to be processed, use the query representation data in the target prediction result that meets the preset reference conditions above to update the tracking query representation data corresponding to the image to be processed, and return to continue executing the above step S1 and its subsequent steps until all image data in the image sequence are traversed.

[0236] The image sequence in step 52 above refers to a sequence that requires target tracking processing; and the present disclosure does not limit the implementation method of the image sequence. For example, it can be implemented using video data.

[0237] The next frame image in the above step 52 refers to image data that exists in the above image sequence, is adjacent to the position of the image to be processed involved in the above step 51, and is located later than the position of the image to be processed involved in the step 51.

[0238] In addition, the present disclosure does not limit the implementation of the above step 52. For example, when the above image sequence is the video data involved in FIG. 2, and the above S1-S3 and step 51 are all used for the image data I shown in FIG. 2, t When processing, the step 52 may specifically be: t After extracting the target prediction result that meets the preset reference conditions from the target description information (for example, the data output by the multi-target tracking model shown in FIG2 ), the image data I t+1 As a new image to be processed, and using the query representation data (Query) in the target prediction result to update the tracking query representation data corresponding to the image to be processed, so that the image data I can be executed based on the updated image to be processed and its corresponding tracking query representation data. t+1 target setting process.

[0239] Based on the relevant content of steps 51 to 52 above, it can be seen that in some application scenarios, after obtaining the target description information of the current frame video image, the target prediction result that meets the preset reference conditions can be extracted from the target description information; then the Query in the target prediction result is used to determine the tracking query representation data corresponding to the next frame video image, so that the tracking query representation data includes the Query, which is conducive to improving the target tracking effect.

[0240] It should be noted that in some application scenarios, if the class confidence of the target prediction result corresponding to a tracking query is low, the query in the target prediction result may be considered invalid. In addition, if the invalidity lasts for more than a predetermined number of frames, the target is considered to have disappeared and will no longer be tracked, and the corresponding tracking query will be deleted.

[0241] It should also be noted that when the image to be processed in the above text refers to an image extracted from an image sequence (for example, video data), if the image to be processed is the first frame image in the image sequence (that is, the first frame image), then since there are no multiple historical images corresponding to the image to be processed, the tracking query representation data corresponding to the image to be processed also does not exist. Therefore, the image processing process for the image to be processed can be specifically: first, feature extraction processing is performed on the image to be processed to obtain a feature extraction result; then, the feature extraction result is encoded to obtain detection query representation data; then, decoding processing is performed based on the detection query representation data to obtain the target description information of the image to be processed; however, if the image to be processed is not the first frame image in the image sequence (for example, the second frame image, the third frame image, etc.), the above S1-S3 can be used to implement the image processing process for the image to be processed.

[0242] Based on the relevant content of the above image processing method, it can be known that the image processing method can be implemented with the help of a pre-trained target tracking model. That is, in a possible implementation, when the target tracking model includes a feature extraction module, an encoding module and a decoding module, the image processing method can be specifically as follows: first, the feature extraction module is used to process the above image to be processed (for example, the image data I shown in FIG2 ). t ) to perform feature extraction processing to obtain feature extraction results; then the encoding module performs encoding processing on the feature extraction results to obtain image feature representation data (for example, the image feature map shown in FIG2 ) and detection query representation data (for example, the D shown in FIG2 ). t ); Finally, the decoding module is based on the image feature representation data, the detection query representation data and the tracking query representation data corresponding to the image to be processed (for example, X shown in FIG. 2 t ) is decoded to obtain target description information of the image to be processed (for example, the data output by the last decoding layer shown in FIG2 ).

[0243] In fact, in order to better improve the performance of the above target tracking model, the present disclosure also provides a possible implementation of the training process of the target tracking model, which may specifically include the following steps 61 and 62.

[0244] Step 61: Utilize at least one second image data and the target detection label of each second image data to train the initial model to obtain the model to be optimized.

[0245] Among them, the second image data refers to the image data required to be used in the first training stage of the target tracking model; and the present disclosure does not limit the method of obtaining the second image data. For example, the second image data can refer to image data extracted from any target detection training data set.

[0246] The target detection label of the second image data is used to describe the actual location of each target in the second image data. Furthermore, the present disclosure does not limit the method for obtaining the target detection label of the second image data; for example, manual labeling can be used. For example, in some application scenarios, if the second image data is image data extracted from a target detection training dataset, the target detection label of the second image data can be label information that exists in the target detection training dataset and has a corresponding relationship with the second image data.

[0247] The initial model is used to represent the target tracking model that needs to be trained in the first training stage; and the present disclosure does not limit the initial model. For example, the initial model may include a feature extraction module, an encoding module, and a decoding model; and the feature extraction module refers to the ResNet50 pre-trained based on the CoCo dataset, and the encoding module refers to the Transformer Encoder module in the DINO detection algorithm.

[0248] The model to be optimized refers to the model obtained by training the initial model for target detection, so that the model to be optimized has better target detection capabilities. It can be seen that the model to be optimized can represent the target tracking model obtained after the first training stage.

[0249] Step 62: Using at least one image sequence and the target tracking label of each image sequence, the decoder in the above model to be optimized is trained to obtain a target tracking model.

[0250] The image sequence in step 62 above refers to the image data sequence required for use in the second training phase of the target tracking model. This disclosure is not limited to this image sequence; for example, it may refer to video data. Furthermore, this disclosure is not limited to the method for obtaining this image sequence; for example, it may refer to video data extracted from any target tracking training dataset.

[0251] The target tracking label of an image sequence is used to describe the actual state of each target within each image data in the image sequence. Furthermore, the present disclosure does not limit the method for obtaining the target tracking label of the image sequence; for example, manual annotation can be used. For example, in some application scenarios, if the image sequence refers to video data extracted from a target tracking training dataset, the target tracking label of the image sequence can be label information that exists in the target tracking training dataset and corresponds to the video data.

[0252] Based on the relevant content of steps 61 to 62 above, it can be seen that in some application scenarios, a two-stage training method can be used to complete the training process for the above target tracking model, and the two-stage training method can be specifically as follows: first, the image-based target detection data is regarded as multi-target tracking training data including video data with a frame length of 1, and the multi-target tracking training data is used to perform the first training stage for the target tracking model; then, the feature extraction module and the encoding module in the trained model are fixed, and the multi-target tracking training data including multiple frames of video data is used to perform the second stage of model training for the decoding module in the model to obtain the final trained target tracking model, so that the target tracking model has better performance.

[0253] In addition, in order to better improve the training effect of the above target tracking model, some data enhancement methods (such as Mosaic and mixup data enhancement methods) can be used in the training process of the above target tracking model. Based on this, it can be seen that the training process of the target tracking model can be specifically as follows: after obtaining at least one second image data, data enhancement processing is first performed on these second image data to obtain enhanced image data; then the enhanced image data and the target detection labels of these enhanced image data are used to train the initial model to obtain the model to be optimized, so that after obtaining the enhanced image sequence obtained by performing data enhancement on at least one image sequence above, the enhanced image sequence and the target tracking label of the enhanced image sequence are used to train the decoder in the above model to be optimized to obtain the target tracking model, which is conducive to improving the training effect of the target tracking model.

[0254] In addition, to further improve the training effect of the target tracking model mentioned above, a multi-scale training method can be used for model training. Based on this, it can be seen that for the target tracking model mentioned above, the scale parameters in the trained model can be randomly adjusted during the training process of the target tracking model to ensure that the trained target tracking model can be applied to image data of any scale, which is conducive to improving the universality of the target tracking model.

[0255] In addition, in order to better improve the model training effect, the present disclosure also provides a process for determining the training loss of the above target tracking model. For ease of understanding, it is explained below with examples.

[0256] As an example, when a certain image sequence is used in the training process of the above target tracking model, and the image sequence includes first image data and multiple historical images corresponding to the first image data, the process of determining the training loss of the target tracking model can specifically include the following steps 71-75.

[0257] Step 71: Determine target description information of the first image data based on the first image data, target description information of multiple historical images corresponding to the first image data, and the above target tracking model.

[0258] The first image data is used to represent the image data used in a round of training. For example, the first image data may be the image data I shown in FIG. t .

[0259] The multiple historical images corresponding to the first image data refer to the image data required for reference when the model training is performed using the first image data. For example, when the first image data is the image data I shown in FIG. t When the first image data corresponds to a plurality of historical images, the plurality of historical images may include the image data I shown in FIG. 2 t-1 , image data I t-2 , image data I t-3 wait.

[0260] The target description information of the first image data is used to describe some targets of the first image data; and the determination process of the "target description information of the first image data" is similar to the determination process of the "target description information of the image to be processed" above. For the sake of brevity, it will not be repeated here.

[0261] Based on the relevant content of step 71 above, it can be seen that for a round of training process, after obtaining the first image data and the target description information of multiple historical images corresponding to the first image data, the target description information of the first image data can be determined based on this information and the target tracking model to be trained, so that the performance of the target tracking model can be analyzed based on the target description information.

[0262] Step 72: Extract the first group of information and the second group of information from the target description information of the first image data.

[0263] Among them, the first group of information refers to the target prediction results that exist in the target description information of the first image data above and meet certain conditions; and the conditions can be set in advance. For example, the conditions can specifically be: the target prediction results corresponding to the detection query or the target prediction results corresponding to the tracking query of the most recent historical image (for example, the query framed by the rectangular dotted box shown in Figure 2).

[0264] The second group of information refers to other target prediction results in the target description information of the first image data except the first group of information (for example, the Query framed by the square dotted box shown in FIG2 ).

[0265] In addition, the present disclosure does not limit the association relationship between the first set of information and the second set of information. For example, the time sequence corresponding to the first set of information is later than the time sequence corresponding to the second set of information. The time sequence corresponding to the first set of information refers to the time sequence of the image data required to predict the first set of information. For example, if the first set of information includes the query framed by the rectangular dotted box shown in Figure 2, then the time sequence corresponding to the first set of information includes the image data I shown in Figure 2. t The timing and image data I shown in FIG2 t-1 The time sequence corresponding to the second set of information refers to the time sequence of the image data required to predict the second set of information. For example, if the second set of information includes the query framed by the square dotted box shown in Figure 2, the time sequence corresponding to the second set of information includes the image data I shown in Figure 2 t-2 The timing of the image data I shown in FIG2 t-3 Timing, etc.

[0266] Based on the line pipe content of step 72 above, it can be known that for a round of training process, after obtaining the target description information of the first image data, the target description information can be split into two groups of information for loss calculation based on the time sequence corresponding to each target prediction result in the target description information. Among them, the first group of information includes the target prediction results corresponding to all detection queries and the target prediction results corresponding to the tracking query of the most recent historical image, and there is no situation in which multiple queries track the same target in the first group of information. The second group of information refers to the target prediction results of other tracking queries after removing the tracking query of the most recent historical image, and there is a situation in which multiple queries track the same target in the second group of information.

[0267] Step 73: Obtain a first loss based on the first set of information and the target label data corresponding to the first image data.

[0268] Among them, the target label data corresponding to the first image data refers to the label information required when using the first image data for model training (for example, the tracking object identification label corresponding to the target, the bounding box label corresponding to the target, the category label corresponding to the target, etc.), so that the target label data can indicate which targets actually exist in the first image data.

[0269] In addition, the present disclosure does not limit the implementation of the above step 73. For example, it can be implemented using any existing or future bipartite matching loss.

[0270] Based on the relevant content of step 73 above, it can be known that for a round of training process, after obtaining the first set of information (for example, the target prediction results corresponding to all detection queries and the target prediction results corresponding to the tracking query of the most recent historical image), there is no situation in the group where multiple queries track the same target, so the binary matching loss can be directly used to perform loss calculation for the first set of information; and the calculation process can be roughly as follows: for the target prediction result corresponding to the tracking query of the tracked target, directly associate the target prediction result corresponding to the query with the target with the tracking object identifier in the target label data corresponding to the first image data above based on the tracking object identifier corresponding to the tracking query, so that the tracking of the tracked target can be achieved. The matching process of the target prediction results corresponding to the query; then, the remaining targets in the target label data that are not associated with any target prediction results are regarded as newly appeared targets, and based on the matching process of bounding boxes and categories, the target prediction results associated with the newly appeared target are determined from the target prediction results corresponding to all detection queries, so that the matching process for the target prediction results corresponding to some detection queries can be achieved; in addition, a new tracking object identifier can be configured for the target prediction results corresponding to those detection queries that have been associated with the newly appeared targets, so that the binary matching loss calculation can be performed based on the target prediction results associated with a target in the target label data to obtain the loss values ​​corresponding to these target prediction results associated with a target in the target label data. In addition, the target prediction results that are not associated with any target in the target label data need to be regarded as the background category, so the category prediction loss corresponding to these target prediction results that are not associated with any target in the target label data can be calculated.

[0271] It should be noted that for the "target prediction result associated with a target in the target label data" in the previous paragraph, the loss value corresponding to the target prediction result can be determined based on the similarity between the target position representation data in the target prediction result and the bounding box label of the associated target, as well as the similarity between the category confidence in the target prediction result and the category label of the associated target.

[0272] It should also be noted that for the above "target prediction result that is not associated with any target in the target label data", the loss value corresponding to the target prediction result can be determined based on the similarity between the category confidence in the target prediction result and the category label corresponding to the background.

[0273] Step 74: Obtain a second loss based on the second set of information and the target label data corresponding to the first image data.

[0274] It should be noted that the present disclosure does not limit the implementation of step 74 above. For example, in some possible implementations, step 74 above may specifically include: determining the second loss based on the label matching result, the second set of information above, and the target label data corresponding to the first image data above. The label matching result is used to describe which target in the target label data corresponding to the first image data each target prediction result in the second set of information is associated with.

[0275] In addition, the present disclosure does not limit the process of determining the above label matching results. For example, it can be implemented by using any existing or future method of associating the target prediction result corresponding to the tracking query of a tracked target with the target in a certain label data (for example, the association method involved in the bipartite matching loss).

[0276] In fact, in some application scenarios, the tracked target described in the second set of information is substantially the same as the tracked target described in the first set of information. Therefore, in order to better save computing resources, the label matching result can be determined based on the matching result between the first set of information and the target label data corresponding to the first image data. For easier understanding, the following example is used to illustrate.

[0277] As an example, assume that the second set of information above includes the target prediction result corresponding to tracking Query1, the first set of information above includes the target prediction result corresponding to tracking Query2, and the tracking object identifier corresponding to tracking Query1 is the same as the tracking object identifier corresponding to tracking Query2. Based on this assumption, it can be known that tracking Query1 and tracking Query2 are used to describe the same tracked target, so based on the matching result between the first set of information above and the target label data corresponding to the first image data, it can be determined which target in the target label data corresponding to the first image data the target prediction result corresponding to tracking Query2 is associated with; and the target associated with the target prediction result corresponding to tracking Query2 can be directly determined as the target associated with the target prediction result corresponding to tracking Query1, so that the purpose of determining the above label matching result based on the matching result between the first set of information and the target label data corresponding to the first image data can be achieved.

[0278] In addition, the present disclosure does not limit the calculation process of the second loss above. For example, when the second set of information above includes at least one target prediction result (for example, the query framed by the square dotted box shown in Figure 2), the calculation process of the second loss may specifically include the following steps 741-742.

[0279] Step 741: Determine the loss corresponding to each target prediction result based on the above label matching result, the above second set of information, and the target label data corresponding to the above first image data.

[0280] It should be noted that the present disclosure is not limited to the implementation method of the above step 741. For example, it can be specifically as follows: first, based on the above label matching result, the above second group of information, and the target label data corresponding to the above first image data, determine the label information corresponding to each target prediction result in the second group of information (for example, bounding box label + category label); then, based on the nth target prediction result in the second group of information and the label information corresponding to the nth target prediction result, determine the loss corresponding to the nth target prediction result (for example, based on the similarity between the target position representation data in the nth target prediction result and the bounding box label in the label information corresponding to the nth target prediction result, and based on the similarity between the category confidence in the nth target prediction result and the category label in the label information corresponding to the nth target prediction result, determine the loss corresponding to the nth target prediction result), where n is a positive integer, n≤N, and N represents the number of target prediction results in the second group of information above.

[0281] Step 742: Determine a second loss based on the average of the losses corresponding to at least one target prediction result in the second set of information.

[0282] Based on the relevant content of step 74 above, it can be known that for the second group of information above, there are multiple queries in the group tracking the same target, so the present disclosure can use tracking consistency loss (tracking object consistency loss) to determine the second loss corresponding to the second group of information. Among them, the tracking consistency loss maintains the same loss calculation method as the above bipartite matching loss, but in the calculation process of the tracking consistency loss, the label information associated with each target prediction result in the second group of information can be directly determined based on the matching result between the first group of information above and the target label data corresponding to the first image data. In addition, the second loss calculated based on the second group of information is averaged within this group, rather than being averaged after being merged with the losses corresponding to all target prediction results in the first group. This can effectively avoid the adverse effects caused by too many target prediction results in the second group of information (for example, the loss determined based on the target prediction result corresponding to the detection query in the first group is diluted), which is conducive to improving the model training effect.

[0283] Step 75: Determine the training loss of the target tracking model based on the first loss and the second loss.

[0284] It should be noted that the present disclosure does not limit the implementation of step 75. For example, step 75 may specifically include: directly determining the sum of the first loss and the second loss as the training loss of the target tracking model. For another example, step 75 may specifically include: performing a weighted sum of the first loss and the second loss to obtain the training loss of the target tracking model. For another example, step 75 may specifically include: combining the first loss and the second loss to obtain the training loss of the target tracking model.

[0285] Based on the relevant contents of steps 71 to 75 above, it can be known that, as shown in Figure 2, during the training process of the target tracking model above, the training loss of the target tracking model can be determined based on the sum of the bipartite matching loss and the tracking consistency loss, so that the training loss can better represent the performance of the model, thereby helping to improve the model training effect.

[0286] It should be noted that the present disclosure does not limit the application scenarios of the training loss shown in steps 71 to 75 above. For example, when the training loss shown in steps 71 to 75 is applied to step 62 above (that is, the second training stage), the "target tracking model" appearing in steps 71 to 75 can be replaced with the model to be optimized above.

[0287] In addition, during the training process of the target tracking model described above, the target prediction results corresponding to the tracking queries involved in the current frame (e.g., the first image data described above) and the target prediction results corresponding to the detection queries associated with the newly appeared targets are all assigned corresponding tracking object identifiers. Therefore, for any target prediction result assigned a tracking object identifier, if it is determined that the IOU between the target prediction result assigned the tracking object identifier and its associated target is greater than a predetermined threshold, it can be determined that the target has been well tracked. Therefore, the query in the target prediction result associated with the target can be sent to the subsequent frame as a historical tracking query (i.e., a positive sample) to participate in tracking according to a preset probability. At the same time, the query in the target prediction result corresponding to the detection query that is not assigned a tracking object identifier and whose target position is closer to the tracking query will be used as a noise tracking query (i.e., a negative sample) with a certain probability, associated with a non-existent target (e.g., a virtual target) and sent to the subsequent frame as a historical tracking query to participate in training, thereby increasing the robustness of the model to noise.

[0288] Based on the relevant content of the target tracking model above, it can be seen that this target tracking model is an end-to-end multi-target tracking algorithm based on multiple queries. It uses multiple historical queries for each target to construct a collaborative tracking query for the target in the current frame, and jointly tracks the target to improve the reliability of the features, so that the target tracking model can achieve better performance in both high frame rate videos and low frame rate videos, and the processing speed is relatively fast.

[0289] Based on the image processing method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide an image processing device, which will be explained and illustrated below in conjunction with Figure 4. Figure 4 is a schematic diagram of the structure of the image processing device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the image processing device provided in the embodiments of the present disclosure, please refer to the relevant content of the image processing method above.

[0290] As shown in FIG4 , an image processing apparatus 400 provided by an embodiment of the present disclosure includes:

[0291] The extraction unit 401 is used to perform feature extraction processing on the image to be processed to obtain a feature extraction result;

[0292] An encoding unit 402 is configured to encode the feature extraction result to obtain detection query representation data;

[0293] The decoding unit 403 is used to perform decoding processing based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0294] In a possible implementation manner, the encoding unit 402 is specifically configured to: perform encoding processing on the feature extraction result to obtain image feature representation data and detection query representation data;

[0295] The decoding unit 403 is specifically configured to perform decoding processing based on the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed, to obtain target description information of the image to be processed.

[0296] In one possible implementation, the multiple historical images include a first category of images and a second category of images; the process of determining the tracking query representation data includes: extracting first tracking target description information from the target description information of the first category of images; and determining the tracking query representation data based on the first tracking target description information and the target description information of the second category of images.

[0297] In one possible implementation, the process of determining the tracking query representation data includes: determining the second tracking target description information based on the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image; and determining the tracking query representation data based on the second tracking target description information and the first tracking target description information.

[0298] In one possible implementation, the image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data; the timing of the image to be processed is later than the timing of the first type of images; and the timing of the first type of images is later than the timing of the second type of images.

[0299] In one possible implementation, the decoding unit 403 is specifically configured to: determine query representation data to be processed based on the detection query representation data and the tracking query representation data; and perform decoding processing based on the query representation data to be processed to obtain target description information of the image to be processed.

[0300] In one possible implementation, the decoding unit 403 is specifically used to: determine the query representation data to be processed based on the detection query representation data and the tracking query representation data; perform decoding processing based on the image feature representation data and the query representation data to be processed to obtain target description information of the image to be processed.

[0301] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained decoder; the decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between the query representation data corresponding to the same target in different historical images.

[0302] In one possible implementation, the at least one target includes one or more targets to be tracked; the at least one decoding layer includes a first decoding layer and a second decoding layer; the decoder also includes a feature fusion module; the feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; the tracking query processing data is determined based on the first decoding layer and the tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each target to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each processed query representation data; the second decoding layer is used to process the tracking query fusion data and the detection query processing data output by the first decoding layer; the detection query processing data is determined based on the first decoding layer and the detection query representation data.

[0303] In one possible implementation, the feature fusion module includes an information removal branch network and an information addition branch network;

[0304] The process of determining the tracking query fusion data includes: using the information removal branch network to process the tracking query processing data to obtain retained information features; using the information addition branch network to process the tracking query processing data to obtain information features to be added; and determining the tracking query fusion data based on the retained information features and the information features to be added.

[0305] In one possible implementation, the process of determining the retained information features includes: performing time series information extraction processing on the tracking query processing data to obtain first time series information; performing self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result; performing full connection processing on the self-attention processing result to obtain a full connection processing result; determining the information representation data to be removed based on the full connection processing result; and performing information removal processing on the tracking query processing data based on the information representation data to be removed to obtain the retained information features.

[0306] In one possible implementation, the process of determining the information features to be added includes: performing time series information extraction processing on the tracking query processing data to obtain second time series information; performing self-attention processing on the second time series information and the tracking query processing data to obtain the information features to be added.

[0307] In one possible implementation, the self-attention layer in the feature fusion module is implemented using a second attention mask, which is used to shield the interaction between query representation data belonging to different targets.

[0308] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0309] The process of determining the training loss of the target tracking model includes: determining the target description information of the first image data based on the first image data, the target description information of multiple historical images corresponding to the first image data, and the target tracking model; extracting a first group of information and a second group of information from the target description information of the first image data; obtaining a first loss based on the first group of information and the target label data corresponding to the first image data; obtaining a second loss based on the second group of information and the target label data; and determining the training loss of the target tracking model based on the first loss and the second loss.

[0310] In a possible implementation manner, the timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

[0311] In one possible implementation, the process of determining the second loss includes: determining the second loss based on a label matching result, the second set of information, and the target label data; the label matching result is determined based on a matching result between the first set of information and the target label data.

[0312] In one possible implementation, the second set of information includes at least one target prediction result;

[0313] The process of determining the second loss includes: determining the loss corresponding to each target prediction result based on the label matching result, the second set of information, and the target label data; and determining the second loss based on the average value of the loss corresponding to the at least one target prediction result.

[0314] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0315] The training process of the target tracking model includes: using at least one second image data and the target detection label of each second image data to train the initial model to obtain the model to be optimized; using at least one image sequence and the target tracking label of each image sequence to train the decoding module in the model to be optimized to obtain the target tracking model.

[0316] In a possible implementation manner, the at least one target includes one or more targets to be tracked; and the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

[0317] In a possible implementation manner, the image to be processed is an image extracted from an image sequence; the target description information includes at least one target prediction result;

[0318] The image processing device 400 further includes:

[0319] A screening unit, configured to extract target prediction results that meet preset reference conditions from the target description information;

[0320] An updating unit is used to update the image to be processed using the next frame image in the image sequence, update the tracking query representation data using the query representation data in the target prediction result that meets the preset reference conditions, and continue to perform the step of feature extraction processing on the image to be processed.

[0321] In a possible implementation manner, the image sequence is video data.

[0322] Based on the relevant content of the above-mentioned image processing device 400, it can be seen that for the image processing device 400 provided in the embodiment of the present disclosure, feature extraction processing is first performed on the image to be processed to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; the feature extraction result is then encoded to obtain detection query representation data (for example, multiple detection queries); finally, decoding processing is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed (for example, the tracking query determined based on multiple historical images) to obtain target description information of the image to be processed (for example, target content feature representation data, target location representation data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed, the tracking query representation data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query representation data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0323] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the image processing method provided by the embodiment of the present disclosure.

[0324] Referring to FIG5 , a schematic diagram of the structure of an electronic device 500 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG5 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0325] As shown in Figure 5, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0326] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0327] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0328] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0329] The embodiments of the present disclosure further provide a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the image processing method provided by the embodiments of the present disclosure.

[0330] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0331] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0332] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0333] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.

[0334] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0335] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0336] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.

[0337] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0338] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0339] It should be noted that the various embodiments of this disclosure are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0340] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0341] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0342] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art (it should be noted that this disclosure is not limited to the implementation of step 33 above).

[0343] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, comprising: Perform feature extraction processing on the image to be processed to obtain a feature extraction result; Encoding the feature extraction result to obtain detection query representation data; Decoding the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; The tracking query representation data is determined based on target description information of a plurality of historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

2. The method according to claim 1, wherein: The plurality of historical images include first-category images and second-category images; The process of determining the tracking query characterization data includes: Extracting first tracking target description information from the target description information of the first type of images; The tracking query representation data is determined according to the first tracking target description information and the target description information of the second type of image.

3. The method according to claim 2, wherein: The determining the tracking query representation data according to the first tracking target description information and the target description information of the second type of image includes: Determine second tracking target description information according to the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image; The tracking query representation data is determined according to the second tracking target description information and the first tracking target description information.

4. The method according to claim 2, wherein: The image to be processed and the multiple historical images corresponding to the image to be processed belong to the same video data; The timing of the to-be-processed images is later than the timing of the first type of images; The timing of the first type of images is later than the timing of the second type of images.

5. The method according to claim 1, wherein: The decoding process is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed, including: Determining query characterization data to be processed according to the detection query characterization data and the tracking query characterization data; Decoding is performed according to the query representation data to be processed to obtain target description information of the image to be processed.

6. The method according to claim 1, wherein: The target description information of the image to be processed is determined using a pre-trained decoder; The decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between corresponding query representation data belonging to the same target in different historical images.

7. The method according to claim 6, wherein: The at least one target includes one or more targets to be tracked; The at least one decoding layer comprises a first decoding layer and a second decoding layer; The decoder also includes a feature fusion module; The feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; The tracking query processing data is determined based on the first decoding layer and the tracking query representation data; The tracking query processing data includes the processed query representation data respectively output by the first decoding layer for each of the to-be-tracked targets; the tracking query fusion data includes the fused query representation data respectively output by the feature fusion module for each of the processed query representation data; The second decoding layer is used for processing the tracking query fusion data and the detection query processing data output by the first decoding layer; The detection query processing data is determined based on the first decoding layer and the detection query representation data.

8. The method according to claim 7, wherein: The feature fusion module includes an information removal branch network and an information addition branch network; The process of determining the tracking query fusion data includes: Processing the tracking query processing data using the information removal branch network to obtain retained information features; Processing the tracking query processing data using the information adding branch network to obtain the information features to be added; Determine the tracking query fusion according to the characteristics of the retained information and the characteristics of the information to be added data.

9. The method according to claim 8, wherein: The process of determining the characteristics of the retained information includes: Performing time series information extraction processing on the tracking query processing data to obtain first time series information; Performing self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result; Performing full connection processing on the self-attention processing result to obtain a full connection processing result; Determining information representation data to be removed according to the full connection processing result; According to the information characterization data to be removed, information removal processing is performed on the tracking query processing data to obtain the retained information characteristics.

10. The method according to claim 8, wherein: The process of determining the information feature to be added includes: Performing time series information extraction processing on the tracking query processing data to obtain second time series information; The second time series information and the tracking query processing data are subjected to self-attention processing to obtain the information features to be added.

11. The method according to claim 7, wherein: The self-attention layer in the feature fusion module is implemented using a second attention mask, which is used to shield the interaction between query representation data belonging to different targets.

12. The method according to claim 1, wherein: The target description information of the image to be processed is determined by using a pre-trained target tracking model; The process of determining the training loss of the target tracking model includes: Determining target description information of the first image data according to the first image data, target description information of a plurality of historical images corresponding to the first image data, and the target tracking model; extracting a first group of information and a second group of information from the target description information of the first image data; Obtaining a first loss according to the first set of information and target label data corresponding to the first image data; Obtaining a second loss according to the second set of information and the target label data; A training loss of the target tracking model is determined based on the first loss and the second loss.

13. The method according to claim 12, wherein: The timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

14. The method according to claim 12, wherein: The obtaining a second loss according to the second set of information and the target label data includes: The second loss is determined according to the label matching result, the second group of information, and the target label data; the label matching result is determined according to the matching result between the first group of information and the target label data.

15. The method according to claim 14, wherein: The second set of information includes at least one target prediction result; The process of determining the second loss includes: Determining the loss corresponding to each of the target prediction results according to the label matching result, the second set of information, and the target label data; The second loss is determined according to an average value of the losses corresponding to the at least one target prediction result.

16. The method according to claim 1, wherein: The target description information of the image to be processed is determined by using a pre-trained target tracking model; The training process of the target tracking model includes: Using at least one second image data and the target detection label of each second image data, training the initial model to obtain a model to be optimized; The target tracking model is obtained by training a decoding module in the model to be optimized using at least one image sequence and a target tracking label of each image sequence.

17. The method according to claim 1, wherein: The at least one target includes one or more targets to be tracked; the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

18. The method according to any one of claims 1 to 17, wherein: The image to be processed refers to an image extracted from an image sequence; the target description information includes at least one target prediction result; After obtaining the target description information of the image to be processed, the method further includes: Extracting target prediction results that meet preset reference conditions from the target description information; The image to be processed is updated using the next frame image in the image sequence, the tracking query characterization data is updated using the query characterization data in the target prediction result that meets the preset reference condition, and the step of performing feature extraction processing on the image to be processed is continued.

19. The method according to claim 18, wherein: The image sequence is video data.

20. An image processing device, comprising: An extraction unit is configured to perform feature extraction processing on the image to be processed to obtain a feature extraction result; An encoding unit, configured to encode the feature extraction result to obtain detection query representation data; A decoding unit is configured to perform decoding processing according to the detection query representation data and the tracking query representation data corresponding to the image to be processed, so as to obtain target description information of the image to be processed; The tracking query representation data is determined based on target description information of a plurality of historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

21. An electronic device, comprising: processor and memory; wherein, The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the method according to any one of claims 1 to 19.

22. A computer readable medium, wherein: The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 19.

Images