Image processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer readable medium, and the method comprises the steps: carrying out the feature extraction processing of a to-be-processed image, and obtaining a feature extraction result; carrying out coding processing on the feature extraction result to obtain image feature representation data and detection query representation data; and finally, according to the image feature representation data, the detection query representation data and the tracking query representation data corresponding to the to-be-processed image, performing decoding processing to obtain target description information of the to-be-processed image, so that the target description information can represent some targets in the to-be-processed image. Wherein the tracking query representation data is determined according to the target description information of the plurality of historical images corresponding to the to-be-processed image, so that the target description information determined based on the tracking query representation data can better represent some targets in the to-be-processed image, and the target tracking effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device, and computer-readable medium. Background Art

[0002] Target tracking is used to analyze the motion state information of some targets from video data (for example, their position in each frame of video image, etc.).

[0003] In addition, for some target tracking implementation schemes (such as a two-stage tracking scheme based on Kalman filtering, etc.), since these implementation schemes are usually only suitable for processing high frame rate videos, when these implementation schemes are used to process low frame rate videos, relatively poor target tracking effects are often presented. Summary of the invention

[0004] The present application provides an image processing method, device, electronic device, and computer-readable medium.

[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:

[0006] The present application provides an image processing method, the method comprising:

[0007] Perform feature extraction processing on the image to be processed to obtain a feature extraction result;

[0008] Encoding the feature extraction result to obtain detection query representation data;

[0009] Decoding is performed based on the detection query characterization data and the tracking query characterization data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query characterization data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0010] In a possible implementation manner, encoding the feature extraction result to obtain detection query representation data includes:

[0011] Encoding the feature extraction result to obtain image feature representation data and detection query representation data;

[0012] The decoding process is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed, including:

[0013] Decoding is performed based on the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed.

[0014] In a possible implementation manner, the plurality of historical images include first-category images and second-category images;

[0015] The process of determining the tracking query characterization data includes:

[0016] Extracting first tracking target description information from the target description information of the first type of images;

[0017] The tracking query representation data is determined according to the first tracking target description information and the target description information of the second type of image.

[0018] In a possible implementation manner, determining the tracking query representation data according to the first tracking target description information and the target description information of the second type of image includes:

[0019] Determine second tracking target description information according to the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image;

[0020] The tracking query representation data is determined according to the second tracking target description information and the first tracking target description information.

[0021] In a possible implementation manner, the image to be processed and a plurality of historical images corresponding to the image to be processed belong to the same video data;

[0022] The timing of the to-be-processed images is later than the timing of the first type of images;

[0023] The timing of the first type of images is later than the timing of the second type of images.

[0024] In a possible implementation manner, the decoding process is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed, including:

[0025] Determining query characterization data to be processed according to the detection query characterization data and the tracking query characterization data;

[0026] Decoding is performed according to the query representation data to be processed to obtain target description information of the image to be processed.

[0027] In a possible implementation manner, the decoding process is performed based on the image feature representation data, the detection query representation data, and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed, including:

[0028] Determining query characterization data to be processed according to the detection query characterization data and the tracking query characterization data;

[0029] Decoding is performed based on the image feature representation data and the query representation data to be processed to obtain target description information of the image to be processed.

[0030] In a possible implementation manner, the target description information of the image to be processed is determined using a pre-trained decoder;

[0031] The decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between corresponding query representation data belonging to the same target in different historical images.

[0032] In a possible implementation manner, the at least one target includes one or more targets to be tracked;

[0033] The at least one decoding layer comprises a first decoding layer and a second decoding layer;

[0034] The decoder also includes a feature fusion module;

[0035] The feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; the tracking query processing data is determined according to the first decoding layer and the tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each of the targets to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each of the processed query representation data;

[0036] The second decoding layer is used to process the tracking query fusion data and the detection query processing data output by the first decoding layer; the detection query processing data is determined according to the first decoding layer and the detection query representation data.

[0037] In a possible implementation manner, the feature fusion module includes an information removal branch network and an information addition branch network;

[0038] The process of determining the tracking query fusion data includes:

[0039] Processing the tracking query processing data using the information removal branch network to obtain retained information features;

[0040] Processing the tracking query processing data using the information adding branch network to obtain the information features to be added;

[0041] The tracking query fusion data is determined according to the retained information features and the to-be-added information features.

[0042] In a possible implementation manner, the process of determining the retained information feature includes:

[0043] Performing time series information extraction processing on the tracking query processing data to obtain first time series information;

[0044] Performing self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result;

[0045] Performing full connection processing on the self-attention processing result to obtain a full connection processing result;

[0046] Determining information representation data to be removed according to the full connection processing result;

[0047] According to the information characterization data to be removed, information removal processing is performed on the tracking query processing data to obtain the retained information characteristics.

[0048] In a possible implementation manner, the process of determining the information feature to be added includes:

[0049] Performing time series information extraction processing on the tracking query processing data to obtain second time series information;

[0050] The second time series information and the tracking query processing data are subjected to self-attention processing to obtain the information features to be added.

[0051] In a possible implementation, the self-attention layer in the feature fusion module is implemented using a second attention mask, where the second attention mask is used to shield the interaction between query representation data belonging to different targets.

[0052] In a possible implementation manner, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0053] The process of determining the training loss of the target tracking model includes:

[0054] Determining target description information of the first image data according to the first image data, target description information of a plurality of historical images corresponding to the first image data, and the target tracking model;

[0055] extracting a first group of information and a second group of information from the target description information of the first image data;

[0056] Obtaining a first loss according to the first set of information and target label data corresponding to the first image data;

[0057] Obtaining a second loss according to the second set of information and the target label data;

[0058] A training loss of the target tracking model is determined based on the first loss and the second loss.

[0059] In a possible implementation manner, the timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

[0060] In a possible implementation manner, obtaining the second loss according to the second set of information and the target label data includes:

[0061] The second loss is determined according to the label matching result, the second group of information, and the target label data; the label matching result is determined according to the matching result between the first group of information and the target label data.

[0062] In a possible implementation manner, the second set of information includes at least one target prediction result;

[0063] The process of determining the second loss includes:

[0064] Determining the loss corresponding to each of the target prediction results according to the label matching result, the second set of information, and the target label data;

[0065] The second loss is determined according to an average value of the losses corresponding to the at least one target prediction result.

[0066] In a possible implementation manner, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0067] The training process of the target tracking model includes:

[0068] Using at least one second image data and the target detection label of each second image data, training the initial model to obtain a model to be optimized;

[0069] The target tracking model is obtained by training a decoding module in the model to be optimized using at least one image sequence and a target tracking label of each image sequence.

[0070] In a possible implementation manner, the at least one target includes one or more targets to be tracked; and the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

[0071] In a possible implementation manner, the image to be processed refers to an image extracted from an image sequence; the target description information includes at least one target prediction result;

[0072] After obtaining the target description information of the image to be processed, the method further includes:

[0073] Extracting target prediction results that meet preset reference conditions from the target description information;

[0074] The image to be processed is updated using the next frame image in the image sequence, the tracking query characterization data is updated using the query characterization data in the target prediction result that meets the preset reference condition, and the step of performing feature extraction processing on the image to be processed is continued.

[0075] In a possible implementation manner, the image sequence is video data.

[0076] The present application provides an image processing device, comprising:

[0077] An extraction unit, used for performing feature extraction processing on the image to be processed to obtain a feature extraction result;

[0078] An encoding unit, used for encoding the feature extraction result to obtain detection query representation data;

[0079] A decoding unit is used to perform decoding processing based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0080] The present application provides an electronic device, the device comprising: a processor and a memory;

[0081] The memory is used to store instructions or computer programs;

[0082] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the image processing method provided in the present application.

[0083] The present application provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes the image processing method provided by the present application.

[0084] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the image processing method provided by the present application.

[0085] Compared with the prior art, this application has at least the following advantages:

[0086] In the technical solution provided by the present application, for an image to be processed (for example, a certain frame of video image in a video data), the image to be processed is first subjected to feature extraction processing to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; the feature extraction result is then encoded to obtain detection query characterization data (for example, multiple detection queries); finally, decoding processing is performed based on the detection query characterization data and the tracking query characterization data corresponding to the image to be processed (for example, the tracking query determined based on multiple historical images) to obtain the target description information of the image to be processed (for example, target content feature characterization data, target location characterization data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query characterization data is determined based on the target description information of multiple historical images corresponding to the image to be processed, the tracking query characterization data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query characterization data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0087] In addition, when the image processing method provided in the present application is used to implement target tracking processing for video data, the image processing method needs to refer to the target description information of multiple historical images (for example, Query) to determine the target in a certain video image, so that in the process of determining the target for the video image, more reference content (for example, the state of the target in different historical images, etc.) can be obtained from the target description information of these historical images. This can effectively overcome the problems caused by the low frame rate of the video data, thereby effectively improving the target tracking effect for low frame rate videos, and thus making the image processing method provided in the present application suitable for target tracking processing for video data with any frame rate, which is conducive to improving the universality of the image processing method in the field of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0089] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0090] Figure 2 A schematic diagram of a target tracking model provided in an embodiment of the present application;

[0091] Figure 3 A schematic diagram of a feature fusion module provided in an embodiment of the present application;

[0092] Figure 4 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0093] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] After research, it was found that for the above two-stage tracking scheme based on Kalman filtering, since the scheme relies on the Kalman filter to model the motion, it requires that the target's motion speed and direction cannot change too quickly. However, since the target in the low frame rate video changes more dramatically between frames (for example, there will be rapid changes in speed, direction, appearance, etc.), the target tracking effect of the scheme in low frame rate video is seriously reduced, making the scheme unsuitable for processing low frame rate video.

[0095] Based on the above research, in order to better improve the target tracking effect, the present application provides an image processing method, which includes: for an image to be processed (for example, a frame of video image in a video data), first performing feature extraction processing on the image to be processed to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; then encoding the feature extraction result to obtain detection query representation data (for example, multiple detection queries); finally, decoding processing is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed (for example, a tracking query determined based on multiple historical images) to obtain target description information of the image to be processed (for example, target content feature representation data, target position representation data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query characterization data is determined based on the target description information of multiple historical images corresponding to the image to be processed, the tracking query characterization data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query characterization data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0096] In addition, when the image processing method provided in the present application is used to implement target tracking processing for video data, the image processing method needs to refer to the target description information of multiple historical images (for example, Query) to determine the target in a certain video image, so that in the process of determining the target for the video image, more reference content (for example, the state of the target in different historical images, etc.) can be obtained from the target description information of these historical images. This can effectively overcome the problems caused by the low frame rate of the video data, thereby effectively improving the target tracking effect for low frame rate videos, and thus making the image processing method provided in the present application suitable for target tracking processing for video data with any frame rate, which is conducive to improving the universality of the image processing method in the field of target tracking.

[0097] In addition, the present application does not limit the execution subject of the image processing method. For example, any implementation of the image processing method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, any implementation of the image processing method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be a stand-alone server, a cluster server or a cloud server.

[0098] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0099] In order to better understand the technical solution provided by the present application, the image processing method provided by the present application is described below with reference to some drawings. Figure 1 As shown, the image processing method provided in the embodiment of the present application includes the following S1-S3. Figure 1 A flowchart of an image processing method provided in an embodiment of the present application.

[0100] S1: Perform feature extraction processing on the image to be processed to obtain feature extraction results.

[0101] The image to be processed refers to the image data that needs to be processed for target determination (for example, Figure 2 The t-th frame video image I shown t ). t is a positive integer. It should be noted that the present application does not limit the implementation method of the target. For example, the target may refer to any detectable object (eg, a person, an animal, a plant, an object, etc.).

[0102] In addition, the present application does not limit the above-mentioned image to be processed. For example, in some application scenarios (such as target tracking processing for a certain image sequence, etc.), the image to be processed refers to an image extracted from the image sequence (for example, image data at the tth arrangement position). The image sequence is used to record multiple image data arranged in a certain order (for example, time sequence); and the present application does not limit the implementation method of the image sequence. For example, it can be implemented using any existing or future image sequence (for example, video data).

[0103] The feature extraction result is used to represent the image information carried by the above image to be processed.

[0104] In addition, the present application does not limit the process of determining the above feature extraction results (that is, the implementation method of S1). For example, it can be implemented by any existing or future method that can perform feature extraction processing on an image data (for example, with the help of a feature extractor, etc.).

[0105] It can be seen that in some application scenarios, S1 above can specifically be: inputting the image to be processed into a pre-constructed feature extractor to obtain a feature extraction result output by the feature extractor, so that the feature extraction result can represent the image information carried by the image to be processed. Among them, the feature extractor is used to perform feature extraction processing on the input data of the feature extractor; and the present application does not limit the implementation method of the feature extractor. For example, the feature extractor can be implemented using any existing or future device with a feature extraction function. For another example, the feature extractor can be implemented using a convolutional neural network (CNN). For another example, the feature extractor can be implemented using the feature extraction module in the target tracking model below.

[0106] It should be noted that the present application does not limit the implementation method of CNN in the above paragraph. For example, it can be implemented using ResNet50 pre-trained based on the CoCo dataset.

[0107] Based on the above two paragraphs, it can be seen that in a possible implementation, when the image processing method provided by the present application uses a pre-trained target tracking model (for example, Figure 2 When the multi-target tracking module shown in the figure is implemented, the above S1 can be specifically: using the feature extraction module in the target tracking model (for example, Figure 2 The CNN shown in the figure performs feature extraction on the above image to be processed to obtain a feature extraction result corresponding to the image to be processed.

[0108] Based on the relevant content of S1 above, it can be known that for the image to be processed (for example, Figure 2 The t-th frame video image I shown t ), after obtaining the image to be processed, feature extraction is performed on the image to be processed to obtain a feature extraction result (for example, Figure 2 The data output by the CNN shown in the figure) is used to make the feature extraction result able to represent the image information carried by the image to be processed, so that some targets in the image to be processed can be determined based on the feature extraction result.

[0109] S2: Encode the feature extraction results to obtain detection query representation data.

[0110] Among them, the detection query characterization data is used to characterize some targets in the above-mentioned image to be processed; and the present application does not limit the detection query characterization data. For example, the detection query characterization data may include query characterization data (for example, Query) corresponding to multiple targets in the image to be processed. Among them, the query characterization data corresponding to the target is used to describe a target in the image to be processed. It can be seen that under a possible implementation method, the detection query characterization data may include query characterization data corresponding to one or more targets under the image to be processed. Among them, the query characterization data corresponding to the target under the image to be processed is used to characterize the state of the target under the image to be processed.

[0111] In addition, the present application does not limit the implementation method of the query representation data corresponding to the target in the above paragraph. For example, it may include target content feature representation data corresponding to the target and target position representation data corresponding to the target. Among them, the target content feature representation data is used to represent the features presented by the target in the image to be processed (for example, features presented in terms of posture, clothing color, etc.); and the present application does not limit the implementation method of the target content feature representation data. For example, the target content feature representation data can be implemented using a feature vector. The target position representation data is used to represent the position of the target in the image to be processed; and the present application does not limit the implementation method of the target content feature representation data. For example, it can be implemented using a bounding box.

[0112] In addition, the present application does not limit the implementation method of the encoding process in S2 above. For example, it can be implemented using any existing or future method that can perform encoding processing (for example, with the help of an encoder, etc.).

[0113] It can be seen that, under a possible implementation, S2 above can specifically be: inputting the above feature extraction results into a pre-built encoder to obtain the detection query representation data output by the encoder. The encoder is used to encode the input data of the encoder; and the present application does not limit the implementation method of the encoder. For example, the encoder can be implemented using any existing or future device with encoding function. For another example, the encoder can be implemented using a Transformer Encoder. For another example, the encoder can be implemented using the encoding module in the target tracking model below.

[0114] It should be noted that the present application does not limit the implementation method of the Transformer Encoder in the above paragraph. For example, it can be implemented using the Transformer Encoder module in the DINO (DETR with Improved DeNoising Anchor Boxes) detection algorithm.

[0115] Based on the above two paragraphs, it can be seen that in a possible implementation, when the image processing method provided by the present application uses a pre-trained target tracking model (for example, Figure 2 When the multi-target tracking module shown in the figure is implemented, the above S2 can be specifically as follows: after determining the feature extraction result from the above image to be processed by the feature extraction module in the target tracking model, the encoding module in the target tracking model performs encoding processing on the feature extraction result to obtain the detection query representation data corresponding to the image to be processed (for example, Figure 2 D t ).

[0116] It should be noted that for Figure 2 D t For example, D t Used to indicate Figure 2 The image data I shown t some targets; and the D t It can be expressed by the following formula (1).

[0117]

[0118] Where D t Indicates that by targeting the above image data I t The query determined by processing (e.g., feature extraction processing + encoding processing); Indicates that the image data I t The query corresponding to the first target in Including target content feature representation data corresponding to the first target and target position representation data corresponding to the first target; Indicates that the image data I t The query corresponding to the second target in Including target content feature representation data corresponding to the second target and target position representation data corresponding to the second target; ... (and so on); Indicates that the image data I t N det The query corresponding to the target, and the Including the Nth dettarget content feature representation data corresponding to the Nth target and the det The target position representation data corresponding to the target; N det Indicates the preset number of queries so that N det To indicate that D t The number of queries involved.

[0119] In fact, in some application scenarios, in order to better improve the target tracking effect, the present application also provides a possible implementation of the encoding process in S2 above. In this implementation, S2 can specifically be: encoding the feature extraction result to obtain image feature representation data and detection query representation data. The image feature representation data is used to represent the information carried by the image to be processed above; and the present application does not limit the implementation of the image feature representation data. For example, it can adopt any existing or future feature encoding vector determined for image data (for example, Figure 2 The implementation is based on the image feature map shown in the figure).

[0120] In addition, the present application does not limit the implementation method of the encoding process in the above paragraph. For example, the encoding process can be specifically: inputting the above feature extraction results into a pre-built encoder to obtain the image feature representation data and the detection query representation data output by the encoder. For another example, when the image processing method provided by the present application uses a pre-trained target tracking model (for example, Figure 2 When the multi-target tracking module shown in the figure is implemented, the encoding process can be specifically as follows: after determining the feature extraction result from the above image to be processed by the feature extraction module in the target tracking model, the encoding module in the target tracking model performs encoding processing on the feature extraction result to obtain image feature representation data corresponding to the image to be processed (for example, Figure 2 The image feature map shown) and the detection query representation data corresponding to the image to be processed (for example, Figure 2 D t ).

[0121] Based on the above content, it can be seen that in a possible implementation mode, for the image to be processed (for example, Figure 2 The t-th frame video image I shown t ), after obtaining the feature extraction result corresponding to the image to be processed (for example, Figure 2 After that, the feature extraction result is encoded to obtain image feature representation data (for example, Figure 2 The image feature map shown) and the detection query representation data (e.g., Figure 2 D t) so that some targets in the image to be processed can be determined based on the image feature characterization data and the detection query characterization data.

[0122] Based on the relevant content of S2 above, it can be seen that for the image to be processed (for example, Figure 2 The t-th frame video image I shown t ), after obtaining the feature extraction result corresponding to the image to be processed (for example, Figure 2 After that, the feature extraction result is encoded to obtain at least the detection query representation data (for example, Figure 2 D t ), so that some targets in the image to be processed can be determined based on the detection query characterization data.

[0123] S3: Decode the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0124] Among them, the tracking query representation data corresponding to the image to be processed refers to the tracking query required to be used as the basis for target determination processing on the image to be processed, so that the tracking query representation data can represent the status of some tracked targets in the image to be processed (for example, the "target to be tracked" below) in the historical stage.

[0125] In addition, the tracking query representation data corresponding to the image to be processed is determined based on the target description information of multiple historical images corresponding to the image to be processed. The time sequence of the historical images is earlier than the time sequence of the image to be processed; and the application does not limit the multiple historical images corresponding to the image to be processed. For example, when the image to be processed is Figure 2 The t-th frame video image I shown t When the multiple historical images corresponding to the image to be processed may include Figure 2 The t-1th frame of video image I shown t -1 , the t-2th frame video image I t-2 , the t-3th frame video image I t-3 The target description information of the historical image refers to the result obtained when the target determination processing is performed on the historical image, so that the target description information of the historical image can describe some targets in the historical image.

[0126] It should be noted that the present application does not limit the implementation method of the target description information. For example, the target description information may include query representation data corresponding to some targets (for example, the feature vector corresponding to the target + the bounding box corresponding to the target). For another example, the target description information may also include the category confidence corresponding to these targets. The category confidence is used to indicate the probability that the target belongs to a certain category.

[0127] It should also be noted that the present application does not limit the implementation method of the “multiple” in the above “multiple historical images corresponding to the image to be processed”. For example, it can be 2, 3 or other positive integers greater than 1.

[0128] In addition, the present application does not limit the implementation method of the above-mentioned process of determining the tracking query representation data corresponding to the image to be processed. For example, when the multiple historical images corresponding to the image to be processed include R image data, the process of determining the tracking query representation data may specifically include: determining the query representation data (Query) corresponding to some or all of the targets existing in the target description information of the r-th image data as the tracking Query corresponding to the r-th image data, where r is a positive integer, r≤R, and R is a positive integer; and then aggregating the tracking Queries corresponding to the R image data to obtain the tracking query representation data, so that the tracking query representation data includes the tracking Queries corresponding to some tracked targets in the multiple historical images corresponding to the image to be processed.

[0129] In fact, in order to better improve the target determination effect, the present application also provides a possible implementation method of the process of determining the tracking query representation data corresponding to the above-mentioned image to be processed. In this implementation method, when the multiple historical images corresponding to the image to be processed include the first type of image (for example, Figure 2 The t-1th frame of video image I shown t-1 ) and the second type of images (e.g. Figure 2 The t-2 frame video image I shown t-2 , the t-3th frame video image I t-3 When the tracking query characterization data corresponding to the image to be processed is determined, the process may include the following steps 11 and 12.

[0130] Step 11: extracting first tracking target description information from the target description information of the first category of images.

[0131] Among them, the first category of images is used to provide the most recent historical position of the target tracked in the above image to be processed; and the present application is not limited to the first category of images. For example, when the image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data, the timing of the image to be processed is later than the timing of the first category of images, and the timing of the first category of images is later than the timing of other images in the multiple historical images except the first category of images (that is, the second category of images above).

[0132] It can be seen that in a possible implementation mode, when the above-mentioned image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data, the above-mentioned first type of image refers to the historical image existing in the multiple historical images and having a time sequence closest to the time sequence of the image to be processed (for example, Figure 2 The t-1th frame of video image I shown t-1 ).

[0133] The first tracking target description information is used to describe the most recent historical information extracted from the first type of image above, which is needed to be referenced when determining some targets in the above image to be processed, so that the first tracking target description information can represent some tracked targets that appear in the historical image whose timing is closest to that of the image to be processed.

[0134] In addition, the present application does not limit the implementation method of the first tracking target description information above. For example, it may include target content feature representation data and target position representation data of at least one tracked target presented in the first type of image above, so that the target content feature representation data can represent the state of the tracked target in the historical image whose time sequence is closest to the time sequence of the image to be processed, and the target position representation data can represent the position of the tracked target in the historical image whose time sequence is closest to the time sequence of the image to be processed.

[0135] In addition, the present application does not limit the determination process of the first tracking target description information above. For example, it may specifically be: after obtaining the target description information of the first category of images above, first extract at least one query that meets the preset filtering conditions from the target description information; then determine the first tracking target description information based on these queries, so that the first tracking target description information includes these queries. Among them, the preset filtering conditions may be set in advance. For example, the preset filtering conditions may be: the query is assigned a tracking object identifier (e.g., ID), and the IOU (Intersection over Union) of the query is greater than a preset threshold. The tracking object identifier is used to uniquely identify the tracking process for a tracked target.

[0136] Based on the relevant content of step 11 above, it can be known that for the above image to be processed (for example, Figure 2 The t-th frame video image I shown t ), after obtaining a historical image (for example, Figure 2 The t-1th frame of video image I shown t-1 ) target description information, some target content feature representation data and target position representation data of the tracked targets can be extracted from the target description information to determine the first tracking target description information (for example, Figure 2 X shown t The tracking Query corresponding to t-1 in the historical image) is used to enable the first tracking target description information to represent the status of the tracked targets in the historical image.

[0137] Step 12: According to the first tracking target description information and the target description information of the second type of image, the tracking query representation data corresponding to the image to be processed is determined.

[0138] Among them, the second category of images is used to represent other images in the multiple historical images corresponding to the above-mentioned image to be processed, except for the above-mentioned first category of images; and the present application is not limited to the second category of images. For example, when the image to be processed and the multiple historical images corresponding to the image to be processed belong to the same video data, the timing of the image to be processed is later than the timing of the first category of images, and the timing of the first category of images is later than the timing of the second category of images.

[0139] It can be seen that in a possible implementation mode, when the above-mentioned image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data, the above-mentioned second type of image refers to the historical image whose time sequence among the multiple historical images is not the closest to the time sequence of the image to be processed (for example, Figure 2 The t-2 frame video image I shown t-2 , the t-3th frame video image I t-3 wait).

[0140] In addition, the present application does not limit the implementation method of step 12 above. For example, it can be specifically as follows: first extract target content feature representation data of some tracked targets from the target description information of the second category of images above; then perform preset processing (for example, integration processing, fusion processing or splicing processing, etc.) on the target content feature representation data and the target content feature representation data of the corresponding targets in the first tracking target description information above, and obtain the tracking query representation data corresponding to the above image to be processed, so that the tracking query representation data can not only represent the positions of these tracked targets in the most recent frame of historical images, but also can more comprehensively represent the characteristics of these tracked targets in the most recent multiple frames of historical images.

[0141] In fact, in order to better improve the target determination effect, the present application also provides another possible implementation of the above step 12, which may specifically include the following steps 121-122.

[0142] Step 121: Determine second tracking target description information according to the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image.

[0143] The second tracking target description information is used to describe the tracking query determined based on the second type of image mentioned above and required to be referenced when determining some targets in the image to be processed mentioned above.

[0144] In addition, the present application does not limit the implementation method of the above step 121. For example, it can be specifically as follows: combining the target position representation data in the above first tracking target description information and the target content feature representation data of the corresponding target in the above second type of image target description information to obtain at least one combined query of the tracked target, so that the combined query can represent the features presented by the tracked target in the second type of image, and the position of the tracked target in the above first type of image; and then determining the second tracking target description information (for example, Figure 2 X shown t The tracking query corresponding to t-2 or the tracking query corresponding to t-3 in the tracking target description information is used to make the second tracking target description information include these combined queries.

[0145] Based on the relevant contents of step 121 above, it can be known that for the above image to be processed (for example, Figure 2 The t-th frame video image I shown t ), when a historical image corresponding to the image to be processed is obtained and is not the closest to the image to be processed (for example, Figure 2 The t-2 frame video image I shownt-2 , or the t-3th frame video image I t-3 After obtaining target description information of the tracked target (such as the target description information), some target content feature representation data of the tracked target can be extracted from the target description information, so that the target content feature representation data can represent the features presented by the tracked target in the historical image; then the target content feature representation data is combined with the target position representation data of the corresponding target in the first tracking target description information above, to obtain a combined Query of the tracked target; finally, based on the combined Query, the second tracking target description information above is determined, so that the second tracking target description information can represent the tracking Query determined based on the historical image and required to be referenced when determining some targets in the image to be processed above.

[0146] Step 122: Determine the tracking query representation data corresponding to the above image to be processed according to the above second tracking target description information and the above first tracking target description information.

[0147] It should be noted that the present application does not limit the implementation of the above step 122. For example, it can be specifically: performing information integration processing on the above second tracking target description information and the above first tracking target description information to obtain the tracking query representation data corresponding to the above image to be processed (for example, Figure 2 X shown t ), so that the tracking query representation data includes some tracking Query sequences corresponding to the tracked targets (for example, Figure 2 The sequence shown sequence And the sequence wait).

[0148] It should also be noted that for Figure 2 The i, i+1, and i+2 shown in the figure respectively represent the tracking object identifier assigned to a tracked target, and i is a positive integer.

[0149] Based on the relevant contents of steps 121 to 122 above, it can be known that in some application scenarios, for the above image to be processed (for example, Figure 2 The t-th frame video image I shown t ), if the multiple historical images corresponding to the image to be processed include the first type of images (for example, Figure 2 The t-1th frame of video image I shown t-1 ) and the second type of images (e.g. Figure 2 The t-2 frame video image I shown t-2 , the t-3th frame video image I t-3etc.), the target position representation data in the target description information of the first category of images and the target content feature representation data in the target description information of the second category of images can be used to construct the tracking Query corresponding to the second category of images (that is, the second tracking target description information above); and then the tracking Query corresponding to the second category of images and the tracking Query corresponding to the first category of images (that is, the first tracking target description information above) are combined to determine the tracking query representation data corresponding to the above image to be processed, so that the tracking query representation data can better represent the status of some tracked targets in the image to be processed in the historical stage.

[0150] Based on the relevant contents of steps 11 to 12 above, it can be known that in some application scenarios, for the above image to be processed (for example, Figure 2 The t-th frame video image I shown t ), we can use the N closest to the image to be processed in the historical time period max Frame history images (e.g. Figure 2 The t-1th frame of video image I shown t-1 , the t-2th frame video image I t -2 , the t-3th frame video image I t-3 etc.), and a historical image frame closest to the image to be processed (for example, Figure 2 The t-1th frame of video image I shown t-1 etc.), construct the tracking query representation data corresponding to the image to be processed (for example, Figure 2 X shown t ), so that all tracking queries recorded in the tracking query representation data can be used to coordinately track some targets in the image to be processed, so that the tracking query representation data can be used to determine some tracked targets that appear in the image to be processed, and the above detection query representation data (for example, Figure 2 D t ) is used to detect newly appeared targets in the image to be processed, so that the target description information of the image to be processed can be determined based on the tracking query characterization data and the detection query characterization data.

[0151] Based on the relevant content of the tracking query representation data corresponding to the above image to be processed, it can be known that in a possible implementation, when there is at least one target in the above image to be processed, if the at least one target includes Y targets to be tracked, then the tracking query representation data includes the query representation data corresponding to the first target to be tracked under multiple historical images, the query representation data corresponding to the second target to be tracked under multiple historical images, ... (and so on), and the query representation data corresponding to the Yth target to be tracked under multiple historical images. Among them, the yth target to be tracked refers to the target that exists in the image to be processed and has been tracked in the historical stage; and the query representation data corresponding to the yth target to be tracked in a historical image is used to represent the state of the yth target to be tracked in the historical image, y is a positive integer, y≤Y, and Y is a positive integer.

[0152] Based on the content of the previous paragraph, it can be known that for the tracking query representation data corresponding to the above-mentioned image to be processed, if the tracking query representation data is used to describe the historical status of one or more tracked targets, then the tracking query representation data includes at least two tracking queries for each tracked target, so that in the subsequent processing process (for example, the decoding process, etc.), for each tracked target, each tracking query corresponding to the target is respectively used as an independent query to complete the tracking prediction processing for the target, and each query will make full use of the information carried in other queries corresponding to the target (for example, target content feature representation data + target position representation data, etc.) to complete its own iterative update processing in this process to achieve collaborative tracking, so that the collaborative tracking process executed for each query is used to jointly track the same target, which can better improve the reliability of the features predicted for the target, so that the features predicted for the target can better express the state of the target in the image to be processed, which is conducive to improving the target tracking effect.

[0153] The above “target description information of the image to be processed” is used to describe at least one target in the image to be processed; and the present application does not limit the implementation method of the “target description information of the image to be processed”. For example, the “target description information of the image to be processed” may include query representation data (Query) of some targets and the category confidence of these targets. Among them, the query representation data of the target may include target content feature representation data of the target and target position representation data of the target (such as a bounding box, etc.). The category confidence of the target is used to indicate how likely the target belongs to certain categories (such as people, animals, cars, etc.).

[0154] For another example, when the detection query representation data corresponding to the above image to be processed includes multiple detection queries, and the tracking query representation data corresponding to the above image to be processed includes multiple tracking queries, the target description information of the image to be processed may include the target prediction results corresponding to each detection query and the target prediction results corresponding to each tracking query. Among them, the target prediction result corresponding to the j-th detection query refers to the state of a target in the image to be processed derived based on the j-th detection query; j is a positive integer, j≤J, J is a positive integer, and J represents the number of the detection query. The target prediction result corresponding to the m-th tracking query refers to the state of a target in the image to be processed derived based on the m-th tracking query; m is a positive integer, m≤M, M is a positive integer, and M represents the number of the tracking query.

[0155] In addition, the present application does not limit the determination process of the above “target description information of the image to be processed” (that is, the implementation method of S3 above), for example, it can be implemented with the aid of a pre-built decoder. The decoder is used to decode the input data of the decoder; and the present application does not limit the decoder, for example, the decoder can be used Figure 2 The decoding module in the multi-target tracking model shown is implemented.

[0156] In addition, in some application scenarios, when the above S2 is implemented by the step of "encoding the feature extraction results to obtain image feature representation data and detection query representation data", the above S3 can be specifically: decoding the image feature representation data, the detection query representation data and the tracking query representation data corresponding to the above image to be processed to obtain the target description information of the image to be processed. It should be noted that the implementation of S3 shown in this paragraph can also be implemented with the help of a pre-built decoder.

[0157] In addition, in order to better improve the target determination effect, the present application also provides a possible implementation of S3 above, which may specifically include the following steps 21-22.

[0158] Step 21: Determine the query representation data to be processed according to the above detection query representation data and the above tracking query representation data.

[0159] Among them, the query representation data to be processed is used to describe all the queries (for example, multiple detection queries + multiple tracking queries) that need to be referenced when performing target determination processing on the above image to be processed, so that the query representation data to be processed can better represent the characteristics of some targets in the image to be processed.

[0160] In addition, the present application does not limit the determination process of the above-mentioned query representation data to be processed. For example, it can specifically be: cascade processing (that is, splicing processing) the above-mentioned detection query representation data and the above-mentioned tracking query representation data to obtain the query representation data to be processed, so that the query representation data to be processed includes the detection query representation data and the tracking query representation data, so that the query representation data to be processed can better represent the characteristics of some targets in the image to be processed.

[0161] Step 22: Decode the query representation data to be processed to obtain target description information of the image to be processed.

[0162] It should be noted that the present application does not limit the implementation of the above step 22. For example, it may specifically be: inputting the above query representation data to be processed into a pre-built decoder (for example, Figure 2 The decoding module in the multi-target tracking model shown in FIG. 1 is used to enable the decoder to perform decoding processing according to the query representation data to be processed, and obtain and output the target description information of the above image to be processed (for example, by Figure 2 The data output by the last decoding layer shown in FIG. 1 ) is used to enable the target description information to describe some targets in the image to be processed.

[0163] For example, in a possible implementation, when the above S2 is implemented by the step of "encoding the feature extraction results to obtain image feature representation data and detection query representation data", the step 22 can be specifically: decoding the image feature representation data and the above query representation data to be processed to obtain the target description information of the image to be processed.

[0164] It should be noted that the present application does not limit the implementation method of the step of "decoding the image feature representation data and the query representation data to be processed to obtain the target description information of the image to be processed" in the above paragraph. For example, it can be specifically: inputting the image feature representation data and the query representation data to be processed into a pre-built decoder (for example, Figure 2 The decoding module in the multi-target tracking model shown in FIG. 1 is used to enable the decoder to perform decoding processing according to the image feature representation data and the query representation data to be processed, and obtain and output the target description information of the above image to be processed (for example, by Figure 2 The data output by the last decoding layer shown in FIG. 1 ) is used to enable the target description information to describe some targets in the image to be processed.

[0165] Based on the relevant content of steps 21 to 22 above, it can be known that in some application scenarios, for the image to be processed, after obtaining the detection query representation data corresponding to the image to be processed and the tracking query representation data corresponding to the image to be processed, the two queries can be cascaded first; and then the cascade results can be directly sent to a pre-built decoder so that the decoder can output the target description information of the image to be processed.

[0166] Based on the above content, it can be seen that in a possible implementation mode, when the image processing method provided by the present application uses a pre-trained target tracking model (for example, Figure 2 When the multi-target tracking module shown in the figure is implemented, the above S3 can be specifically as follows: for the above image to be processed (for example, Figure 2 The t-th frame video image I shown t ), after obtaining the tracking query representation data corresponding to the image to be processed (for example, Figure 2 X shown t ), the detection query representation data corresponding to the image to be processed output by the encoding module in the target tracking model (for example, Figure 2 D t ) and the image feature representation data corresponding to the image to be processed (for example, Figure 2 After that, the tracking query representation data and the detection query representation data can be cascaded; then the decoding module in the target tracking model performs decoding processing based on the cascade result and the image feature representation data, so that the decoding module can obtain and output the target description information of the image to be processed by autoregression (for example, by Figure 2 The data output by the last decoding layer shown in FIG. 1 ) is used to make the target description information include the query of some targets and the category confidence of these targets.

[0167] Based on the above content, it can be known that in a possible implementation mode, the target description information of the above image to be processed is determined by using a pre-trained decoder. The decoder can be used to perform decoding processing based on the above detection query representation data and the tracking query representation data corresponding to the above image to be processed to obtain the target description information of the image to be processed; or, the decoder can also be used to perform decoding processing based on the above image feature representation data, the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed.

[0168] In addition, in order to better improve the target determination effect, the present application also provides a possible implementation of the above decoder, under which the decoder may include at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between the query representation data corresponding to the same target in different historical images, so that the decoding layer can achieve the effect of temporal blocking. It should be noted that the present application does not limit the number of the decoding layers, for example, it can be 6.

[0169] It should be noted that the present application does not limit the implementation method of the "query representation data corresponding to the same target in different historical images" in the previous paragraph. For example, when the decoder shown in the previous paragraph refers to the h-th decoder, the "query representation data corresponding to the same target in different historical images" refers to the different tracking queries corresponding to the same target existing in the input data of the self-attention module in the h-th decoder; h is a positive integer, h≤the number of decoders (for example, 6).

[0170] Based on the content of the above paragraph, it can be known that in one possible implementation, the above decoder can be composed of a plurality of (for example, 6) temporal blocking decoding layers (temporal blocking decoder) stacked together, and the self-attention modules in each temporal blocking decoding layer are implemented using the first attention mask, so that each self-attention module can use the first attention mask to shield the interaction between the query representation data corresponding to the same target in different historical images, so as to avoid mutual inhibition between different tracking queries of the same target.

[0171] The first attention mask mentioned above refers to the attention mask configured for the self-attention module in the decoding layer above, so that the self-attention module can use the attention mask to shield the interaction between the query representation data corresponding to the same target in different historical images; and the first attention mask is configured to shield the interaction between the query representation data corresponding to the same target in different historical images. In addition, the present application does not limit the implementation method of the first attention mask, for example, Figure 2 The first attention mask in the self-attention module in the decoding layer shown can be taken as follows

[0172] The attention masks shown in Table 1 are implemented.

[0173]

[0174] Table 1 A possible implementation of the attention mask in the self-attention module in the decoding layer

[0175] It should be noted that for Table 1 above, Indicates that the target identified by i as the tracking object is in the image data I t-1 The corresponding tracking query is as follows; Indicates that the target with i+1 as the tracking object identifier is in the image data I t -1 The corresponding tracking query is as follows; Indicates that the target with i+1 as the tracking object identifier is in the image data I t-2 The corresponding tracking query is as follows; Indicates that the target with i+1 as the tracking object identifier is in the image data I t-3 The corresponding tracking Query below; … (and so on).

[0176] Based on the relevant content of the above decoder, it can be known that in a possible implementation, the above decoder can be stacked by multiple time-blocked decoding layers, and the attention mask in the self-attention module in each time-blocked decoding layer can be configured to shield the interaction between different tracking queries belonging to the same target, so as to avoid mutual inhibition between different tracking queries for the same target. For example, for the self-attention module in the decoding layer, the self-attention module can be used not only for interactive processing between queries of different targets, but also for interactive processing between the same query; however, the self-attention module will shield the interactive processing between different tracking queries under the same target.

[0177] It can be seen that, since the various input information of the self-attention module can interact equally in the traditional self-attention module, in order to effectively avoid the suppression of multiple tracking queries of the same target by each other, a mask (for example, the first attention mask above or the mask shown in Table 1 above) can be added to the self-attention module to obtain the self-attention module in the decoding layer above, so that the self-attention module in the decoding layer is the self-attention module with the mask added. Among them, because the interaction between different tracking queries belonging to the same target can be shielded in the self-attention module with the mask added, it can effectively avoid the phenomenon that multiple tracking queries of the same target suppress each other in the self-attention module with the mask added, which is beneficial to improve the target tracking effect.

[0178] In fact, in order to better improve the decoding effect, the present application also provides another possible implementation of the above decoder. In this implementation, when the decoder includes at least one decoding layer, and the at least one decoding layer includes a first decoding layer and a second decoding layer, the decoder may also include a feature fusion module (for example, Figure 2The feature fusion module shown in FIG. 1 ). The first decoding layer is used to represent a decoding layer located in front of any two adjacent decoding layers in the at least one decoding layer. The second decoding layer is used to represent a decoding layer located in the back of any two adjacent decoding layers in the at least one decoding layer. The feature fusion module is located between the first decoding layer and the second decoding layer. For ease of understanding, the following is explained in conjunction with an example.

[0179] As an example, when the above decoder includes 6 decoding layers arranged in sequence, if the first decoding layer is the 1st decoding layer in the decoder, the second decoding layer is the 2nd decoding layer in the decoder, and there is a feature fusion module between the 1st decoding layer and the 2nd decoding layer; if the first decoding layer is the 2nd decoding layer in the decoder, the second decoding layer is the 3rd decoding layer in the decoder, and there is a feature fusion module between the 2nd decoding layer and the 3rd decoding layer; ... (and so on).

[0180] Based on the above two paragraphs, it can be known that for any triplet (first decoding layer, feature fusion module, second decoding layer) existing in the above decoder, the association relationship between different elements in the triplet can be specifically: the feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; and the second decoding layer is used to process the tracking query fusion data output by the feature fusion module and the detection query processing data output by the first decoding layer. Among them, the tracking query processing data is determined based on the first decoding layer and the above tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each target to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each processed query representation data; the detection query processing data is determined based on the first decoding layer and the above detection query representation data.

[0181] The tracking query processing data mentioned above refers to the Query output by the first decoding layer mentioned above for the tracked target; and the tracking query processing data is determined based on the first decoding layer and the tracking query representation data corresponding to the image to be processed mentioned above. For example, if the first decoding layer is the first decoding layer in the above decoder, the tracking query processing data output by the first decoding layer refers to the Query obtained by processing the above tracking query representation data by the first decoding layer, so that the tracking query processing data output by the first decoding layer includes the processed query representation data output by the first decoding layer for each tracked target; if the first decoding layer is the second decoding layer in the above decoder, the tracking query processing data output by the first decoding layer refers to the Query obtained by processing the Query used to describe the tracked target in the input data of the second decoding layer by the second decoding layer. Query to which the tracking query processing data output by the first decoding layer includes the processed query representation data output by the second decoding layer for each tracked target; if the first decoding layer is the third decoding layer in the decoder above, the tracking query processing data output by the first decoding layer refers to the query obtained by the third decoding layer processing the query for describing the tracked target in the input data of the third decoding layer, so that the tracking query processing data output by the first decoding layer includes the processed query representation data output by the third decoding layer for each tracked target; ... (and so on). Wherein, the processed query representation data refers to the query iterated by the first decoding layer for a tracked target.

[0182] The above detection query processing data refers to the Query output by the above first decoding layer for the target to be detected; and the detection query processing data is determined based on the first decoding layer and the above detection query representation data. For example, if the first decoding layer is the first decoding layer in the above decoder, the detection query processing data output by the first decoding layer refers to the Query obtained by the first decoding layer processing the above detection query representation data; if the first decoding layer is the second decoding layer in the above decoder, the detection query processing data output by the first decoding layer refers to the Query obtained by the second decoding layer processing the Query for describing the target to be detected in the input data of the second decoding layer; if the first decoding layer is the third decoding layer in the above decoder, the detection query processing data output by the first decoding layer refers to the Query obtained by the third decoding layer processing the Query for describing the target to be detected in the input data of the third decoding layer; ... (and so on).

[0183] The tracking query fusion data mentioned above refers to the fusion result obtained by fusing the query associated with each tracked target output by the first decoding layer mentioned above by the feature fusion module above, so that the tracking query fusion data includes the fused query representation data output by the feature fusion module for each tracked target respectively, so that the tracking query fusion data can better represent the characteristics presented by each tracked target. Among them, the fused query representation data refers to the fusion result obtained by fusing the tracked target and the query associated with the tracked target. For example, for the tracked target A, if the target A corresponds to 3 tracking query processing data Query A-1 , Query A-2 , Query A-3 , then in this Query A-1 In the corresponding fusion process, the Query A-1 With Query A-2 , Query A-3 The information carried is interacted with to obtain the Query A-1 The corresponding fused query representation data Query A-1 '; In this Query A-2 In the corresponding fusion process, the Query A-2 With Query A-1 , Query A-3 The information carried is interacted with to obtain the Query A-2 The corresponding fused query representation data Query A-2 '; In this Query A-3 In the corresponding fusion process, the Query A-3 With Query A-1 , Query A-2 The information carried is interacted with to obtain the Query A-3 The corresponding fused query representation data Query A-3 '.

[0184] In addition, the present application does not limit the implementation method of the feature fusion module above. For example, in some application scenarios, the feature fusion module may include multiple fusion submodules, and different fusion submodules are used to perform fusion processing on tracking queries associated with different tracked targets. It can be seen that the number of fusion submodules can be determined according to the number of tracked targets. For example, in another possible implementation, the feature fusion module can be used to perform the following steps: fusing the tracking query associated with the first tracked target to obtain and output the fused query representation data corresponding to the first tracked target, so that the fused query representation data corresponding to the first tracked target includes the fusion results corresponding to each tracking query associated with the first tracked target, so that the number of queries in the fused query representation data corresponding to the first tracked target is consistent with the number of tracking queries associated with the first tracked target; fusing the tracking query associated with the second tracked target to obtain and output the fused query representation data corresponding to the second tracked target, so that the fused query representation data corresponding to the second tracked target includes the fusion results corresponding to each tracking query associated with the second tracked target, so that the number of queries in the fused query representation data corresponding to the second tracked target is consistent with the number of tracking queries associated with the second tracked target; ... (and so on).

[0185] In fact, in order to better improve the fusion efficiency, the present application also provides a possible implementation method of the above-mentioned feature fusion module. Under this implementation method, the feature fusion module includes an information removal branch network and an information addition branch network; and the process of using the feature fusion module to determine the above-mentioned tracking query fusion data can specifically include the following steps 31-33.

[0186] Step 31: Use the above information removal branch network to process the above tracking query processing data to obtain the retained information features.

[0187] Among them, the information removal branch network is used to perform information removal processing on the input data of the above feature fusion module; and the present application does not limit the information removal branch network. For example, the information removal branch network may include two self-attention layers, a fully connected layer and a sigmoid layer.

[0188] The retained information feature is used to represent the information retained after information removal processing is performed on the input data of the above feature fusion module.

[0189] In addition, the present application does not limit the determination process of the above-mentioned retained information features. For example, it may specifically include the following steps 311 to 315.

[0190] Step 311: Perform timing information extraction processing on the above tracking query processing data to obtain first timing information, so that the first timing information can represent the timing information carried in the tracking query processing data.

[0191] It should be noted that the present application does not limit the implementation method of step 311. For example, step 311 can be implemented with the help of a self-attention layer.

[0192] Based on the relevant content of step 311 above, it can be known that for the above information, the branch network is removed (for example, Figure 3 The information removal branch network shown in FIG. 1 is a network in which the information removal branch network receives the tracking query processing data output by the first decoding layer above (for example, Figure 3 shown ), the first self-attention layer in the information removal branch network can perform time series information extraction processing on the tracking query processing data to obtain the first time series information (for example, Figure 3 The timing information 1) shown is used so that the first timing information can represent the timing information carried in the tracking query processing data.

[0193] It should be noted that for Figure 3 shown For example, the Can be expressed by Figure 2 The first decoding layer in Figure 2 X shown t the results of the processing; and F t Used to describe the feature information predicted by the first decoding layer for some tracked targets; Represents the position information predicted by the first decoding layer for some tracked targets.

[0194] Step 312: Perform self-attention processing on the above first time series information and the above tracking query processing data to obtain a self-attention processing result.

[0195] It should be noted that the present application does not limit the implementation method of step 312. For example, step 312 can be implemented with the help of a self-attention layer.

[0196] Based on the relevant content of step 312 above, it can be known that for the above information, the branch network is removed (for example, Figure 3For the information removal branch network shown in FIG. 1 , the first self-attention layer in the information removal branch network outputs the first temporal information above (for example, Figure 3 After the timing information 1) shown, the second self-attention layer in the branch network can be removed by the information to perform self-attention processing based on the first timing information and the above tracking query processing data to obtain a self-attention processing result, so that the self-attention processing result can indicate which information needs to be eliminated.

[0197] Step 313: Perform full connection processing on the above self-attention processing result to obtain a fully connected processing result.

[0198] It should be noted that the present application does not limit the implementation method of step 313. For example, step 313 can be implemented with the aid of a fully connected layer.

[0199] Based on the relevant content of step 313 above, it can be known that for the above information, the branch network is removed (for example, Figure 3 For the information removal branch network shown in the figure, after the second self-attention layer in the information removal branch network outputs the above self-attention processing result, the fully connected layer in the information removal branch network can perform fully connected processing on the self-attention processing result to obtain a fully connected processing result.

[0200] Step 314: Determine the information representation data to be removed according to the above full-connection processing result, so that the information representation data to be removed can indicate which information in the above tracking query processing data needs to be removed.

[0201] It should be noted that the present application does not limit the implementation method of step 314. For example, step 314 can be implemented with the help of a sigmoid layer.

[0202] Based on the relevant content of step 314 above, it can be known that for the above information, the branch network is removed (for example, Figure 3 For example, after the fully connected layer in the information removal branch network outputs the above fully connected processing result, the sigmoid layer in the information removal branch network can process the fully connected processing result to obtain the information representation data to be removed (for example, Figure 3 Z) as shown, so that the to-be-removed information characterization data can indicate which information in the above tracking query processing data needs to be removed.

[0203] Step 315: Based on the above information characterization data to be removed, information removal processing is performed on the above tracking query processing data to obtain the retained information features.

[0204] It should be noted that the present application does not limit the implementation of step 315. For example, step 315 may specifically include: firstly determining the information representation data to be retained based on the above information representation data to be removed (for example, Figure 3 1-Z as shown); then multiply the target content feature representation data in the above tracking query processing data by the information representation data to be retained to obtain the retained information feature (for example, Figure 3 F t ×(1-Z)).

[0205] Based on the relevant contents of steps 311 to 315 above, it can be known that in a possible implementation mode, the above information removal branch network may include two self-attention layers, a fully connected layer and a sigmoid layer; wherein the first self-attention layer is used to collect time series information, the second self-attention layer is used to fuse the features of multiple queries, and the fully connected layer is used to perform fully connected processing on the output data of the second self-attention layer; the sigmoid layer is used to process the output data of the fully connected layer to obtain a gating value (that is, the above information representation data to be removed) so that it can be subsequently based on the gating value (for example, Figure 3 Z) shown, information removal processing is performed on the input data of the information removal branch network.

[0206] Based on the relevant content of step 31 above, it can be known that for the above feature fusion module, if the feature fusion module includes an information removal branch network, the tracking query processing data (for example, Figure 3 shown ), the information removal branch network performs information removal processing on the tracking query processing data to obtain the retained information features (for example, Figure 3 F t ×(1-Z)), so as to subsequently determine the query fusion result for the tracking query processing data based on the retained information feature.

[0207] Step 32: Use the above information to add a branch network to process the above tracking query processing data to obtain the information features to be added.

[0208] Among them, the information adding branch network is used to extract the required added information from the input data of the above feature fusion module; and the present application does not limit the information adding branch network, for example, the information adding branch network can include two self-attention layers.

[0209] The feature of the information to be added refers to the information required as a basis for the information addition process.

[0210] In addition, the present application does not limit the process of determining the information features to be added above. For example, it may specifically include the following steps 321 and 322.

[0211] Step 321: Perform timing information extraction processing on the above tracking query processing data to obtain second timing information, so that the second timing information can represent the timing information carried in the tracking query processing data.

[0212] It should be noted that the present application does not limit the implementation method of step 321. For example, step 321 can be implemented with the aid of a self-attention layer.

[0213] Based on the relevant content of step 321 above, it can be known that a branch network is added to the above information (for example, Figure 3 For example, the information adding branch network shown in FIG. 1 receives the tracking query processing data output by the first decoding layer above (for example, Figure 3 shown ), the first self-attention layer in the information adding branch network can perform time series information extraction processing on the tracking query processing data to obtain the second time series information (for example, Figure 3 The timing information 2 shown is used to enable the second timing information to represent the timing information carried in the tracking query processing data.

[0214] Step 322: Perform self-attention processing on the second time series information and the tracking query processing data to obtain the information features to be added.

[0215] It should be noted that the present application does not limit the implementation method of step 322. For example, step 322 can be implemented with the help of a self-attention layer.

[0216] Based on the relevant content of step 322 above, it can be known that for the above information, the branch network is removed (for example, Figure 3 For example, the information adding branch network shown in FIG. 1 is used to add the second temporal information above (for example, Figure 3 After the timing information 2 shown in the figure is obtained, the second self-attention layer in the branch network can be removed by the information, and the self-attention processing is performed according to the second timing information and the above tracking query processing data to obtain the information feature to be added (for example, Figure 3 shown ) so that the feature of the information to be added can represent the information required for the information adding process.

[0217] Based on the relevant contents of steps 321 to 322 above, it can be known that in one possible implementation, the above information adding branch network may include two self-attention layers; wherein the first self-attention layer is used to collect time series information, and the second self-attention layer is used to fuse the features of multiple queries.

[0218] Based on the relevant content of step 32 above, it can be known that for the feature fusion module above, if the feature fusion module includes an information adding branch network, then the tracking query processing data (for example, Figure 3 shown ) After that, the information adding branch network performs information extraction processing on the tracking query processing data to obtain the information features to be added (for example, Figure 3 shown ), so as to subsequently determine the query fusion result for the tracking query processing data based on the information feature to be added.

[0219] Step 33: Determine the above tracking query fusion data according to the above retained information features and the above to-be-added information features.

[0220] It should be noted that the present application does not limit the implementation method of the above step 33. For example, it can be implemented using the following formula (2).

[0221]

[0222] In the formula, represents the output result of the feature fusion module above (for example, the tracking query fusion data above); LN() represents the hierarchical normalization processing; F t represents the target content feature representation data in the input data of the feature fusion module above; Z represents the gate value output by the sigmoid layer in the information removal branch network in the feature fusion module above; F t ×(1-Z) represents the data output by the information removal branch network in the feature fusion module above (for example, the information retention feature above); Represents the data output by the second self-attention layer in the information adding branch network in the feature fusion module above (for example, the information features to be added above).

[0223] Based on the relevant contents of steps 31 to 33 above, it can be known that for the feature fusion module above, the feature fusion module can include two branch networks, namely, an information removal branch network and an information addition branch network. Each branch network includes two superimposed self-attention layers, the first self-attention layer is used to collect time series information, and the second self-attention layer is used to fuse the features of multiple queries according to the time series information. In addition, for the information removal branch network, the output data of the second self-attention layer in the information removal branch network will further pass through a fully connected layer and a sigmoid layer to obtain a gated value, and the gated value and the output data of the information addition branch network jointly determine the fused query information (as shown in formula (2) above). It can be seen that because the feature fusion module simulates the information interaction process through two branches, so that for a certain tracking query of any target, the feature fusion module can more efficiently integrate the valid information carried by other tracking queries of the target except the certain tracking query into the certain tracking query, so as to complete the self-iterative update for the certain tracking query, which is conducive to improving the interaction efficiency and effect between different tracking queries of the same target, thereby improving the target tracking effect.

[0224] In addition, for the above feature fusion module, in order to ensure that the feature fusion module can simultaneously perform query fusion processing for all tracked targets, the present application also provides a possible implementation of the feature fusion module. In this implementation, the self-attention layer in the feature fusion module is implemented using a second attention mask, and the second attention mask is used to shield the interaction between query representation data belonging to different targets. This ensures that only interactions between all tracking queries of the same target are performed in the feature fusion module.

[0225] The second attention mask refers to the attention mask configured for the self-attention layer in the feature fusion module above; and the second attention mask is configured to shield the interaction between query representation data belonging to different targets. In addition, the present application does not limit the implementation of the second attention mask, for example, Figure 2 The second attention mask in the self-attention layer in the feature fusion module shown can be implemented using the attention mask shown in Table 2 below.

[0226] It should be noted that for each branch network in the above feature fusion module (for example, an information removal branch network or an information addition branch network), since the branch network includes two self-attention layers, in one possible implementation, the second attention mask used in each self-attention layer in the branch network can be implemented using an attention mask similar to that shown in Table 2 below.

[0227] It should also be noted that for the following Table 2, the as well as See above for more information.

[0228]

[0229] Table 2 A possible implementation of the attention mask in the self-attention layer in the feature fusion module

[0230] Based on the relevant content of the feature fusion module above, it can be known that the feature fusion module can be used to simultaneously perform query fusion processing for all tracked targets; and the attention mask in the self-attention layer in the feature fusion module can be configured to avoid interaction between queries of different targets, so as to ensure that only interactions between queries of the same target are performed within the feature fusion module.

[0231] In addition, for each branch network in the above feature fusion module (for example, an information removal branch network or an information addition branch network), since the branch network includes two self-attention layers, the second attention mask used by the g-th self-attention layer in the branch network is used to shield the interaction between the query representation data of different targets appearing in the input data of the g-th self-attention layer, where g is a positive integer, g≤2.

[0232] Based on the above decoder content, it can be seen that in one possible implementation, the decoder can be composed of a stack of multiple time-blocked decoding layers, and a feature fusion module will be inserted between any two adjacent decoding layers, so as to ensure that the decoder has a better decoding effect.

[0233] Based on the relevant contents of S1 to S3 above, it can be known that for the image processing method provided in the embodiment of the present application, the image to be processed is first subjected to feature extraction processing to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; the feature extraction result is then encoded to obtain at least detection query characterization data (e.g., multiple detection queries); finally, at least the detection query characterization data and the tracking query characterization data corresponding to the image to be processed (e.g., the tracking query determined based on multiple historical images) are decoded to obtain the target description information of the image to be processed (e.g., target content feature characterization data, target location characterization data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query characterization data is determined based on the target description information of multiple historical images corresponding to the image to be processed, so that the tracking query characterization data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query characterization data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0234] In fact, in some application scenarios, after obtaining the target description information of an image data, some high-quality target prediction results can be selected from the target description information as the target determination results of the image data. Based on this, the present application also provides a possible implementation of the above image processing method. Under this implementation, when the target description information of the above image to be processed includes at least one target prediction result (for example, target prediction results corresponding to multiple detection queries + target prediction results corresponding to multiple tracking queries), the image processing method, in addition to the above S1-S3, may also include the following step 41. The execution time of step 41 is later than the execution time of S3 above.

[0235] Step 41: After obtaining the target description information of the image to be processed, extract the target prediction result that meets the preset reference condition from the target description information to obtain the target determination result of the image to be processed.

[0236] Among them, the preset reference condition refers to the condition required to screen high-quality target prediction results from the target description information of the image to be processed above; and the present application does not limit the preset reference condition, for example, it can be: the target prediction result corresponding to the detection query or the target prediction result corresponding to the tracking query of the most recent historical image, and the category confidence reaches the preset threshold.

[0237] The target determination result of the image to be processed is used to describe some targets in the image to be processed that are predicted more accurately.

[0238] In addition, the present application does not limit the determination process of the target determination result of the above-mentioned image to be processed. For example, it can be specifically as follows: after obtaining the target description information of the image to be processed, the target prediction result corresponding to the detection query and the target prediction result corresponding to the tracking query of the most recent historical image are extracted from the target description information, and both are used as the prediction results to be processed; then, it is determined whether the category confidence in each prediction result to be processed reaches a preset threshold. If so, it can be determined that the target described by the prediction result to be processed is relatively accurate. Therefore, the target determination result of the image to be processed can be determined based on the prediction result to be processed, so that the target determination result includes the prediction result to be processed.

[0239] Based on the relevant content of step 41 above, it can be known that in some application scenarios, after determining the target description information of an image data, some high-quality target prediction results are extracted from the target description information to obtain the target determination result of the image data, so that the target determination result can better represent which targets exist in the image data, which is conducive to improving the target determination effect.

[0240] In fact, when the image processing method provided by the embodiment of the present application is applied to the field of target tracking, after obtaining the target description information of a certain frame of image data, the target description information can be further used to determine the tracking query representation data corresponding to the next frame of image, which is conducive to improving the target determination effect for the next frame of image. Based on this, the present application also provides a possible implementation of the above image processing method. Under this implementation, when the above image to be processed refers to an image extracted from an image sequence (for example, video data), and the target description information of the image to be processed includes at least one target prediction result (for example, target prediction results corresponding to multiple detection queries + target prediction results corresponding to multiple tracking queries), the image processing method may include at least S1-S3 above and steps 51-52 below. Among them, the execution time of step 51 is later than the execution time of S3.

[0241] Step 51: After obtaining the target description information of the image to be processed, extracting the target prediction result that meets the preset reference condition from the target description information.

[0242] It should be noted that the relevant content of step 51 is similar to part of the content of step 41 above, and for the sake of brevity, it will not be repeated here.

[0243] Step 52: Use the next frame image in the above image sequence to update the image to be processed, use the query representation data in the target prediction result that meets the preset reference condition above to update the tracking query representation data corresponding to the image to be processed, and return to continue executing the above step S1 and its subsequent steps until all image data in the image sequence are traversed.

[0244] The image sequence in step 52 above refers to a sequence that needs to be processed for target tracking; and the present application does not limit the implementation method of the image sequence, for example, it can be implemented using video data.

[0245] The next frame image in the above step 52 refers to image data existing in the above image sequence, adjacent to the position of the image to be processed involved in the above step 51, and later than the position of the image to be processed involved in the step 51.

[0246] In addition, the present application does not limit the implementation of the above step 52. For example, when the above image sequence is Figure 2 The video data involved, and the above S1-S3 and step 51 are all used for Figure 2 The image data I shown t When processing, the step 52 may specifically be: t Extract the target prediction results that meet the preset reference conditions from the target description information (for example, Figure 2 After that, the image data I t+1 As a new image to be processed, and using the query representation data (Query) in the target prediction result to update the tracking query representation data corresponding to the image to be processed, so that the image data I can be executed based on the updated image to be processed and its corresponding tracking query representation data. t+1 target setting process.

[0247] Based on the relevant contents of steps 51 to 52 above, it can be known that in some application scenarios, after obtaining the target description information of the current frame video image, the target prediction result that meets the preset reference conditions can be extracted from the target description information; and then the Query in the target prediction result is used to determine the tracking query representation data corresponding to the next frame video image, so that the tracking query representation data includes the Query, which is conducive to improving the target tracking effect.

[0248] It should be noted that in some application scenarios, for a target prediction result corresponding to a tracking query, if the category confidence in the target prediction result is low, the query in the target prediction result can be determined to be invalid. In addition, if the invalidity lasts for more than a predetermined number of frames, the target is considered to have disappeared and will no longer be tracked, and the corresponding tracking query will be deleted.

[0249] It should also be noted that, when the image to be processed in the above text refers to an image extracted from an image sequence (for example, video data), if the image to be processed is the first frame image in the image sequence (that is, the first frame image), then since there are no multiple historical images corresponding to the image to be processed, the tracking query representation data corresponding to the image to be processed also does not exist. Therefore, the image processing process for the image to be processed can be specifically: first, feature extraction processing is performed on the image to be processed to obtain a feature extraction result; then, the feature extraction result is encoded to obtain detection query representation data; then, decoding processing is performed based on the detection query representation data to obtain the target description information of the image to be processed; however, if the image to be processed is not the first frame image in the image sequence (for example, the second frame image, the third frame image, etc.), the above S1-S3 can be used to implement the image processing process for the image to be processed.

[0250] Based on the above content of the image processing method, it can be known that the image processing method can be implemented with the help of a pre-trained target tracking model. That is, in a possible implementation, when the target tracking model includes a feature extraction module, an encoding module and a decoding module, the image processing method can be specifically as follows: first, the feature extraction module is used to extract the image to be processed (for example, Figure 2 The image data I shown t ) to perform feature extraction processing to obtain feature extraction results; then the encoding module performs encoding processing on the feature extraction results to obtain image feature representation data (for example, Figure 2 The image feature map shown) and the detection query representation data (e.g., Figure 2 D t ); Finally, the decoding module is based on the image feature representation data, the detection query representation data and the tracking query representation data corresponding to the image to be processed (for example, Figure 2 X shown t ) to decode and obtain the target description information of the image to be processed (for example, Figure 2 The data output by the last decoding layer shown).

[0251] In fact, in order to better improve the performance of the above target tracking model, the present application also provides a possible implementation of the training process of the target tracking model, which may specifically include the following steps 61-62.

[0252] Step 61: Train the initial model using at least one second image data and the target detection label of each second image data to obtain a model to be optimized.

[0253] Among them, the second image data refers to the image data required to be used in the first training stage of the target tracking model; and the present application does not limit the method of obtaining the second image data. For example, the second image data may refer to image data extracted from any target detection training data set.

[0254] The target detection label of the second image data is used to describe the actual location of each target in the second image data; and the present application does not limit the method for obtaining the target detection label of the second image data, for example, it can be implemented by means of manual annotation. For example, in some application scenarios, if the second image data refers to image data extracted from a target detection training data set, then the target detection label of the second image data can be label information existing in the target detection training data set and having a corresponding relationship with the second image data.

[0255] The initial model is used to represent the target tracking model that needs to be trained in the first training stage; and this application does not limit the initial model. For example, the initial model may include a feature extraction module, an encoding module, and a decoding model; and the feature extraction module refers to the ResNet50 pre-trained based on the CoCo dataset, and the encoding module refers to the Transformer Encoder module in the DINO detection algorithm.

[0256] The model to be optimized refers to the model obtained by training the initial model above for target detection, so that the model to be optimized has a better target detection function. It can be seen that the model to be optimized can represent the target tracking model obtained after the first training stage.

[0257] Step 62: Using at least one image sequence and the target tracking label of each image sequence, the decoder in the above model to be optimized is trained to obtain a target tracking model.

[0258] The image sequence in step 62 above refers to the image data sequence required to be used in the second training phase for the target tracking model; and the present application does not limit the image sequence, for example, it can refer to video data. In addition, the present application does not limit the acquisition method of the image sequence, for example, the image sequence can refer to video data extracted from any target tracking training data set.

[0259] The target tracking label of the image sequence is used to describe the actual state of each target in each image data in the image sequence; and the present application does not limit the method of obtaining the target tracking label of the image sequence, for example, it can be implemented by means of manual annotation. For example, in some application scenarios, if the image sequence refers to video data extracted from a target tracking training data set, the target tracking label of the image sequence can be the label information existing in the target tracking training data set and having a corresponding relationship with the video data.

[0260] Based on the relevant contents of steps 61 to 62 above, it can be known that in some application scenarios, a two-stage training method can be used to complete the training process for the above target tracking model, and the two-stage training method can be specifically as follows: first, the image-based target detection data is regarded as multi-target tracking training data including video data with a frame length of 1, and the multi-target tracking training data is used to perform the first training stage for the target tracking model; then, the feature extraction module and the encoding module in the trained model are fixed, and the multi-target tracking training data including multiple frames of video data is used to perform the second stage of model training for the decoding module in the model to obtain the final trained target tracking model, so that the target tracking model has better performance.

[0261] In addition, in order to better improve the training effect of the above target tracking model, some data enhancement methods (such as Mosaic and mixup data enhancement methods) can be used in the training process of the above target tracking model. Based on this, it can be seen that the training process of the target tracking model can be specifically as follows: after acquiring at least one second image data, data enhancement processing is first performed on these second image data to obtain enhanced image data; then the enhanced image data and the target detection labels of these enhanced image data are used to train the initial model to obtain the model to be optimized, so that after obtaining the enhanced image sequence obtained by performing data enhancement on at least one image sequence above, the enhanced image sequence and the target tracking label of the enhanced image sequence are used to train the decoder in the above model to be optimized to obtain the target tracking model, which is conducive to improving the training effect of the target tracking model.

[0262] In addition, in order to better improve the training effect of the above target tracking model, a multi-scale training method can be used for model training. Based on this, it can be seen that for the above target tracking model, the scale parameters in the trained model can be randomly adjusted during the training process of the target tracking model to ensure that the finally trained target tracking model can be applied to process image data with any scale, which is conducive to improving the universality of the target tracking model.

[0263] In addition, in order to better improve the model training effect, the present application also provides a process for determining the training loss of the above target tracking model. For ease of understanding, it is explained below with examples.

[0264] As an example, when a certain image sequence is used in the training process of the above target tracking model, and the image sequence includes first image data and multiple historical images corresponding to the first image data, the process of determining the training loss of the target tracking model may specifically include the following steps 71-75.

[0265] Step 71: Determine target description information of the first image data based on the first image data, target description information of a plurality of historical images corresponding to the first image data, and the above target tracking model.

[0266] The first image data is used to represent the image data used in a round of training. For example, the first image data can be Figure 2 The image data I shown t .

[0267] The multiple historical images corresponding to the first image data refer to the image data required for reference when the first image data is used for model training. Figure 2 The image data I shown t When the first image data corresponds to a plurality of historical images, the plurality of historical images may include Figure 2 The image data I shown t-1 、Image data I t-2 、Image data I t-3 wait.

[0268] The target description information of the first image data is used to describe some targets of the first image data; and the determination process of the "target description information of the first image data" is similar to the determination process of the "target description information of the image to be processed" above. For the sake of brevity, it will not be repeated here.

[0269] Based on the relevant content of step 71 above, it can be known that for a round of training process, after obtaining the first image data and the target description information of multiple historical images corresponding to the first image data, the target description information of the first image data can be determined based on this information and the target tracking model that needs to be trained, so that the performance of the target tracking model can be analyzed based on the target description information.

[0270] Step 72: Extract the first group of information and the second group of information from the target description information of the first image data.

[0271] The first group of information refers to the target prediction results that exist in the target description information of the first image data above and meet certain conditions; and the condition can be preset, for example, the condition can specifically be: the target prediction result corresponding to the detection query or the target prediction result corresponding to the tracking query of the most recent historical image (for example, Figure 2 Query is shown as being framed by a dotted rectangular box).

[0272] The second set of information refers to other target prediction results (for example, Figure 2 Query is shown as being framed by a square dashed box).

[0273] In addition, the present application does not limit the association relationship between the first set of information and the second set of information, for example, the timing corresponding to the first set of information is later than the timing corresponding to the second set of information. The timing corresponding to the first set of information refers to the timing of the image data required to predict the first set of information. For example, if the first set of information includes Figure 2 The query framed by the dotted rectangle shown in the figure, then the time series corresponding to the first set of information includes Figure 2 The image data I shown t The timing and Figure 2 The image data I shown t-1 The time sequence corresponding to the second set of information refers to the time sequence of the image data required to be used when predicting the second set of information. For example, if the second set of information includes Figure 2 The query framed by the square dashed box shown in the figure, then the time series corresponding to the second set of information includes Figure 2 The image data I shown t-2 The timing of Figure 2 The image data I shown t-3 The timing of .

[0274] Based on the line pipe content of step 72 above, it can be known that for a round of training process, after obtaining the target description information of the first image data, the target description information can be split into two groups of information for loss calculation based on the time sequence corresponding to each target prediction result in the target description information. Among them, the first group of information includes the target prediction results corresponding to all detection queries and the target prediction results corresponding to the tracking query of the most recent historical image, and there is no situation in which multiple queries track the same target in the first group of information. The second group of information refers to the target prediction results of other tracking queries after removing the tracking query of the most recent historical image, and there is a situation in which multiple queries track the same target in the second group of information.

[0275] Step 73: Obtain a first loss based on the first set of information and the target label data corresponding to the first image data.

[0276] Among them, the target label data corresponding to the first image data refers to the label information required when using the first image data for model training (for example, the tracking object identification label corresponding to the target, the bounding box label corresponding to the target, the category label corresponding to the target, etc.), so that the target label data can indicate which targets actually exist in the first image data.

[0277] In addition, the present application does not limit the implementation of the above step 73. For example, it can be implemented by using any existing or future bipartite matching loss.

[0278] Based on the relevant content of step 73 above, it can be known that for a round of training process, after obtaining the first group of information (for example, the target prediction results corresponding to all detection queries and the target prediction results corresponding to the tracking query of the most recent historical image), there is no situation in the group where multiple queries track the same target, so the binary matching loss can be directly used to perform loss calculation for the first group of information; and the calculation process can be roughly as follows: for the target prediction result corresponding to the tracking query of the tracked target, directly associate the target prediction result corresponding to the query with the target with the tracking object identifier in the target label data corresponding to the first image data above based on the tracking object identifier corresponding to the tracking query, so that the tracking of the tracked target can be achieved. The target prediction results corresponding to the query are matched; then, the remaining targets in the target label data that are not associated with any target prediction results are regarded as newly appeared targets, and based on the matching process of bounding boxes and categories, the target prediction results associated with the newly appeared target are determined from the target prediction results corresponding to all detection queries, so that the matching process for the target prediction results corresponding to some detection queries can be realized; in addition, a new tracking object identifier can be configured for the target prediction results corresponding to the detection query that have been associated with the newly appeared target, so that the target prediction results associated with a target in the target label data can be used to perform binary matching loss calculations, and the loss values ​​corresponding to the target prediction results associated with a target in the target label data can be obtained. In addition, the target prediction results that are not associated with any target in the target label data need to be regarded as the background category, so the category prediction loss corresponding to these target prediction results that are not associated with any target in the target label data can be calculated.

[0279] It should be noted that, for the "target prediction result associated with a target in the target label data" in the previous paragraph, the loss value corresponding to the target prediction result can be determined based on the similarity between the target position representation data in the target prediction result and the bounding box label of the associated target, and the similarity between the category confidence in the target prediction result and the category label of the associated target.

[0280] It should also be noted that for the above "target prediction result that is not associated with any target in the target label data", the loss value corresponding to the target prediction result can be determined based on the similarity between the category confidence in the target prediction result and the category label corresponding to the background.

[0281] Step 74: Obtain a second loss based on the second set of information and the target label data corresponding to the first image data.

[0282] It should be noted that the present application does not limit the implementation of the above step 74. For example, in some possible implementations, the above step 74 may specifically include: determining the second loss based on the label matching result, the above second set of information, and the target label data corresponding to the above first image data. The label matching result is used to describe which target in the target label data corresponding to the first image data each target prediction result in the second set of information is associated with.

[0283] In addition, the present application does not limit the process of determining the above label matching results. For example, it can be implemented by any existing or future method of associating the target prediction result corresponding to the tracking query of a tracked target with the target in a certain label data (for example, the association method involved in the binary matching loss).

[0284] In fact, in some application scenarios, the tracked target described in the second set of information above is roughly the same as the tracked target described in the first set of information above, so in order to better save computing resources, the label matching result above can be determined based on the matching result between the first set of information above and the target label data corresponding to the first image data. For ease of understanding, the following is an example.

[0285] As an example, assume that the second set of information above includes the target prediction result corresponding to tracking Query1, the first set of information above includes the target prediction result corresponding to tracking Query2, and the tracking object identifier corresponding to the tracking Query1 is the same as the tracking object identifier corresponding to the tracking Query2. Based on this assumption, it can be known that the tracking Query1 and the tracking Query2 are used to describe the same tracked target, so it is possible to determine which target in the target label data corresponding to the first image data the target prediction result corresponding to the tracking Query2 is associated with based on the matching result between the first set of information above and the target label data corresponding to the first image data; and directly determine the target associated with the target prediction result corresponding to the tracking Query2 as the target associated with the target prediction result corresponding to the tracking Query1, so that the purpose of determining the above label matching result based on the matching result between the first set of information and the target label data corresponding to the first image data can be achieved.

[0286] In addition, the present application does not limit the calculation process of the second loss above. For example, when the second set of information above includes at least one target prediction result (for example, Figure 2 When the Query is framed by a square dotted box as shown, the calculation process of the second loss may specifically include the following steps 741-742.

[0287] Step 741: Determine the loss corresponding to each target prediction result based on the above label matching result, the above second group of information, and the target label data corresponding to the above first image data.

[0288] It should be noted that the present application does not limit the implementation method of step 741 above. For example, it can be specifically as follows: first, based on the above label matching result, the above second group of information, and the target label data corresponding to the above first image data, determine the label information corresponding to each target prediction result in the second group of information (for example, bounding box label + category label); then, based on the nth target prediction result in the second group of information and the label information corresponding to the nth target prediction result, determine the loss corresponding to the nth target prediction result (for example, based on the similarity between the target position representation data in the nth target prediction result and the bounding box label in the label information corresponding to the nth target prediction result, and based on the similarity between the category confidence in the nth target prediction result and the category label in the label information corresponding to the nth target prediction result, determine the loss corresponding to the nth target prediction result), where n is a positive integer, n≤N, and N represents the number of target prediction results in the second group of information above.

[0289] Step 742: Determine a second loss based on the average value of the losses corresponding to at least one target prediction result in the second set of information above.

[0290] Based on the relevant content of step 74 above, it can be known that for the second group of information above, there are multiple queries in the group tracking the same target, so the present application can use the tracking consistency loss (tracking object consistency loss) to determine the second loss corresponding to the second group of information. Among them, the tracking consistency loss maintains the same loss calculation method as the above binary matching loss, but in the calculation process of the tracking consistency loss, the label information associated with each target prediction result in the second group of information can be directly determined based on the matching result between the first group of information above and the target label data corresponding to the first image data. In addition, the second loss calculated based on the second group of information is averaged within this group, rather than being averaged after the loss corresponding to all target prediction results in the first group is merged, so that the adverse effects caused by too many target prediction results in the second group of information can be effectively avoided (for example, the loss determined based on the target prediction result corresponding to the detection query in the first group is diluted, etc.), which is conducive to improving the model training effect.

[0291] Step 75: Determine the training loss of the target tracking model based on the first loss and the second loss.

[0292] It should be noted that the present application does not limit the implementation method of step 75. For example, step 75 may specifically be: directly determining the sum of the first loss and the second loss as the training loss of the target tracking model. For another example, step 75 may specifically be: performing weighted summation of the first loss and the second loss to obtain the training loss of the target tracking model. For another example, step 75 may specifically be: combining the first loss and the second loss to obtain the training loss of the target tracking model.

[0293] Based on the relevant contents of steps 71 to 75 above, it can be seen that if Figure 2 As shown, during the training process of the target tracking model above, the training loss of the target tracking model can be determined according to the sum of the bipartite matching loss and the tracking consistency loss, so that the training loss can better represent the performance of the model, which is beneficial to improving the model training effect.

[0294] It should be noted that the present application does not limit the application scenarios of the training loss shown in steps 71 to 75 above. For example, when the training loss shown in steps 71 to 75 is applied to step 62 above (that is, the second training stage), the "target tracking model" appearing in steps 71 to 75 can be replaced with the model to be optimized above.

[0295] In addition, in the training process of the target tracking model described above, the target prediction results corresponding to the tracking queries involved in the current frame (for example, the first image data described above) and the target prediction results corresponding to the detection queries associated with the newly appeared targets are all assigned corresponding tracking object identifiers. Therefore, for any target prediction result assigned with a tracking object identifier, if it is determined that the IOU between the target prediction result assigned with the tracking object identifier and the target associated with it is greater than a predetermined threshold, it can be determined that the target has been well tracked. Therefore, the query in the target prediction result associated with the target can be sent to the subsequent frame as a historical tracking query (that is, a positive sample) to participate in tracking according to a preset probability. At the same time, the query in the target prediction result corresponding to the detection query that is not assigned with a tracking object identifier and whose target position is close to the tracking query will be used as a noise tracking query (that is, a negative sample) with a certain probability, associated with a non-existent target (for example, a virtual target) and sent to the subsequent frame as a historical tracking query to participate in training, thereby increasing the robustness of the model to noise.

[0296] Based on the relevant content of the target tracking model above, it can be known that this target tracking model is an end-to-end multi-target tracking algorithm based on multiple queries. It uses multiple historical queries of each target to construct a collaborative tracking query for the target in the current frame, and jointly tracks the target to improve the reliability of the features, so that the target tracking model can achieve better performance in both high frame rate videos and low frame rate videos, and the processing speed is relatively fast.

[0297] Based on the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing device. Figure 4 Explain and illustrate. Figure 4 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. It should be noted that for the technical details of the image processing device provided in an embodiment of the present application, please refer to the relevant contents of the image processing method above.

[0298] like Figure 4 As shown, the image processing device 400 provided in the embodiment of the present application includes:

[0299] An extraction unit 401 is used to perform feature extraction processing on the image to be processed to obtain a feature extraction result;

[0300] An encoding unit 402 is used to encode the feature extraction result to obtain detection query representation data;

[0301] The decoding unit 403 is used to perform decoding processing based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query representation data is determined based on the target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

[0302] In a possible implementation manner, the encoding unit 402 is specifically used to: encode the feature extraction result to obtain image feature representation data and detection query representation data;

[0303] The decoding unit 403 is specifically used to: perform decoding processing according to the image feature representation data, the detection query representation data and the tracking query representation data corresponding to the image to be processed, so as to obtain the target description information of the image to be processed.

[0304] In one possible implementation, the multiple historical images include a first category of images and a second category of images; the process of determining the tracking query representation data includes: extracting first tracking target description information from the target description information of the first category of images; and determining the tracking query representation data based on the first tracking target description information and the target description information of the second category of images.

[0305] In one possible implementation, the process of determining the tracking query characterization data includes: determining the second tracking target description information based on the target position characterization data in the first tracking target description information and the target content feature characterization data in the target description information of the second type of image; and determining the tracking query characterization data based on the second tracking target description information and the first tracking target description information.

[0306] In a possible implementation, the image to be processed and multiple historical images corresponding to the image to be processed belong to the same video data; the timing of the image to be processed is later than the timing of the first category of images; the timing of the first category of images is later than the timing of the second category of images.

[0307] In a possible implementation, the decoding unit 403 is specifically used to: determine the query representation data to be processed based on the detection query representation data and the tracking query representation data; and perform decoding processing based on the query representation data to be processed to obtain target description information of the image to be processed.

[0308] In one possible implementation, the decoding unit 403 is specifically used to: determine the query representation data to be processed based on the detection query representation data and the tracking query representation data; perform decoding processing based on the image feature representation data and the query representation data to be processed to obtain target description information of the image to be processed.

[0309] In one possible implementation, the target description information of the image to be processed is determined using a pre-trained decoder; the decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between the query representation data corresponding to the same target in different historical images.

[0310] In one possible implementation, the at least one target includes one or more targets to be tracked; the at least one decoding layer includes a first decoding layer and a second decoding layer; the decoder also includes a feature fusion module; the feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; the tracking query processing data is determined based on the first decoding layer and the tracking query representation data; the tracking query processing data includes the processed query representation data output by the first decoding layer for each of the targets to be tracked; the tracking query fusion data includes the fused query representation data output by the feature fusion module for each of the processed query representation data; the second decoding layer is used to process the tracking query fusion data and the detection query processing data output by the first decoding layer; the detection query processing data is determined based on the first decoding layer and the detection query representation data.

[0311] In a possible implementation manner, the feature fusion module includes an information removal branch network and an information addition branch network;

[0312] The process of determining the tracking query fusion data includes: using the information removal branch network to process the tracking query processing data to obtain the retained information characteristics; using the information addition branch network to process the tracking query processing data to obtain the information characteristics to be added; determining the tracking query fusion data based on the retained information characteristics and the information characteristics to be added.

[0313] In one possible implementation, the process of determining the retained information features includes: performing timing information extraction processing on the tracking query processing data to obtain first timing information; performing self-attention processing on the first timing information and the tracking query processing data to obtain a self-attention processing result; performing full connection processing on the self-attention processing result to obtain a full connection processing result; determining information representation data to be removed based on the full connection processing result; and performing information removal processing on the tracking query processing data based on the information representation data to be removed to obtain the retained information features.

[0314] In one possible implementation, the process of determining the information features to be added includes: performing timing information extraction processing on the tracking query processing data to obtain second timing information; performing self-attention processing on the second timing information and the tracking query processing data to obtain the information features to be added.

[0315] In a possible implementation, the self-attention layer in the feature fusion module is implemented using a second attention mask, where the second attention mask is used to shield the interaction between query representation data belonging to different targets.

[0316] In a possible implementation manner, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0317] The process of determining the training loss of the target tracking model includes: determining the target description information of the first image data based on the first image data, the target description information of multiple historical images corresponding to the first image data, and the target tracking model; extracting a first group of information and a second group of information from the target description information of the first image data; obtaining a first loss based on the first group of information and the target label data corresponding to the first image data; obtaining a second loss based on the second group of information and the target label data; and determining the training loss of the target tracking model based on the first loss and the second loss.

[0318] In a possible implementation manner, the timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

[0319] In one possible implementation, the process of determining the second loss includes: determining the second loss based on a label matching result, the second set of information, and the target label data; the label matching result is determined based on a matching result between the first set of information and the target label data.

[0320] In a possible implementation manner, the second set of information includes at least one target prediction result;

[0321] The process of determining the second loss includes: determining the loss corresponding to each of the target prediction results based on the label matching result, the second set of information, and the target label data; and determining the second loss based on the average value of the loss corresponding to at least one target prediction result.

[0322] In a possible implementation manner, the target description information of the image to be processed is determined using a pre-trained target tracking model;

[0323] The training process of the target tracking model includes: using at least one second image data and the target detection label of each second image data to train the initial model to obtain the model to be optimized; using at least one image sequence and the target tracking label of each image sequence to train the decoding module in the model to be optimized to obtain the target tracking model.

[0324] In a possible implementation manner, the at least one target includes one or more targets to be tracked; and the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

[0325] In a possible implementation manner, the image to be processed refers to an image extracted from an image sequence; the target description information includes at least one target prediction result;

[0326] The image processing device 400 further includes:

[0327] A screening unit, used to extract target prediction results that meet preset reference conditions from the target description information;

[0328] An updating unit is used to update the image to be processed using the next frame image in the image sequence, update the tracking query characterization data using the query characterization data in the target prediction result that meets the preset reference condition, and continue to execute the step of performing feature extraction processing on the image to be processed.

[0329] In a possible implementation manner, the image sequence is video data.

[0330] Based on the relevant contents of the above-mentioned image processing device 400, it can be known that for the image processing device 400 provided in the embodiment of the present application, the image to be processed is first subjected to feature extraction processing to obtain a feature extraction result, so that the feature extraction result is used to characterize the image information carried by the image to be processed; the feature extraction result is then encoded to obtain detection query characterization data (e.g., multiple detection queries); finally, decoding processing is performed based on the detection query characterization data and the tracking query characterization data corresponding to the image to be processed (e.g., the tracking query determined based on multiple historical images) to obtain the target description information of the image to be processed (e.g., target content feature characterization data, target location characterization data, and category confidence, etc.), so that the target description information can represent some targets in the image to be processed. Among them, because the tracking query characterization data is determined based on the target description information of multiple historical images corresponding to the image to be processed, so that the tracking query characterization data can better represent the historical status of some targets in the image to be processed, so that the target description information determined based on the tracking query characterization data can better represent some targets in the image to be processed, which is conducive to improving the target tracking effect.

[0331] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the image processing method provided in the embodiment of the present application.

[0332] See also Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0333] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0334] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0335] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0336] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0337] The embodiment of the present application also provides a computer-readable medium, in which instructions or computer programs are stored. When the instructions or computer programs are executed on a device, the device executes any implementation of the image processing method provided in the embodiment of the present application.

[0338] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0339] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0340] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0341] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.

[0342] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0343] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0344] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.

[0345] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0346] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0347] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0348] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0349] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0350] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0351] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: Perform feature extraction processing on the image to be processed to obtain a feature extraction result; Encoding the feature extraction result to obtain detection query representation data; Decoding is performed based on the detection query characterization data and the tracking query characterization data corresponding to the image to be processed to obtain target description information of the image to be processed; the tracking query characterization data is determined based on target description information of multiple historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

2. The method according to claim 1, characterized in that The plurality of historical images include first-category images and second-category images; The process of determining the tracking query characterization data includes: Extracting first tracking target description information from the target description information of the first type of images; The tracking query representation data is determined according to the first tracking target description information and the target description information of the second type of image.

3. The method according to claim 2, characterized in that The determining the tracking query representation data according to the first tracking target description information and the target description information of the second type of image includes: Determine second tracking target description information according to the target position representation data in the first tracking target description information and the target content feature representation data in the target description information of the second type of image; The tracking query representation data is determined according to the second tracking target description information and the first tracking target description information.

4. The method according to claim 2, characterized in that: The image to be processed and the multiple historical images corresponding to the image to be processed belong to the same video data; The timing of the to-be-processed images is later than the timing of the first type of images; The timing of the first type of images is later than the timing of the second type of images.

5. The method according to claim 1, characterized in that The decoding process is performed based on the detection query representation data and the tracking query representation data corresponding to the image to be processed to obtain the target description information of the image to be processed, including: Determining query characterization data to be processed according to the detection query characterization data and the tracking query characterization data; Decoding is performed according to the query representation data to be processed to obtain target description information of the image to be processed.

6. The method according to claim 1, characterized in that The target description information of the image to be processed is determined using a pre-trained decoder; The decoder includes at least one decoding layer, and the self-attention module in the decoding layer is implemented using a first attention mask, which is used to shield the interaction between corresponding query representation data belonging to the same target in different historical images.

7. The method according to claim 6, characterized in that The at least one target includes one or more targets to be tracked; The at least one decoding layer comprises a first decoding layer and a second decoding layer; The decoder also includes a feature fusion module; The feature fusion module is used to perform fusion processing on the tracking query processing data output by the first decoding layer to obtain tracking query fusion data; The tracking query processing data is determined based on the first decoding layer and the tracking query representation data; The tracking query processing data includes the processed query representation data respectively output by the first decoding layer for each of the to-be-tracked targets; the tracking query fusion data includes the fused query representation data respectively output by the feature fusion module for each of the processed query representation data; The second decoding layer is used for processing the tracking query fusion data and the detection query processing data output by the first decoding layer; The detection query processing data is determined based on the first decoding layer and the detection query representation data.

8. The method according to claim 7, characterized in that The feature fusion module includes an information removal branch network and an information addition branch network; The process of determining the tracking query fusion data includes: Processing the tracking query processing data using the information removal branch network to obtain retained information features; Processing the tracking query processing data using the information adding branch network to obtain the information features to be added; The tracking query fusion data is determined according to the retained information features and the to-be-added information features.

9. The method according to claim 8, characterized in that The process of determining the characteristics of the retained information includes: Performing time series information extraction processing on the tracking query processing data to obtain first time series information; Performing self-attention processing on the first time series information and the tracking query processing data to obtain a self-attention processing result; Performing full connection processing on the self-attention processing result to obtain a full connection processing result; Determining information representation data to be removed according to the full connection processing result; According to the information characterization data to be removed, information removal processing is performed on the tracking query processing data to obtain the retained information characteristics.

10. The method according to claim 8, characterized in that The process of determining the information feature to be added includes: Performing time series information extraction processing on the tracking query processing data to obtain second time series information; The second time series information and the tracking query processing data are subjected to self-attention processing to obtain the information features to be added.

11. The method according to claim 7, characterized in that The self-attention layer in the feature fusion module is implemented using a second attention mask, which is used to shield the interaction between query representation data belonging to different targets.

12. The method according to claim 1, characterized in that The target description information of the image to be processed is determined by using a pre-trained target tracking model; The process of determining the training loss of the target tracking model includes: Determining target description information of the first image data according to the first image data, target description information of a plurality of historical images corresponding to the first image data, and the target tracking model; extracting a first group of information and a second group of information from the target description information of the first image data; Obtaining a first loss according to the first set of information and target label data corresponding to the first image data; Obtaining a second loss according to the second set of information and the target label data; A training loss of the target tracking model is determined based on the first loss and the second loss.

13. The method according to claim 12, characterized in that The timing corresponding to the first group of information is later than the timing corresponding to the second group of information.

14. The method according to claim 12, characterized in that The obtaining a second loss according to the second set of information and the target label data includes: The second loss is determined according to the label matching result, the second group of information, and the target label data; the label matching result is determined according to the matching result between the first group of information and the target label data.

15. The method according to claim 14, characterized in that The second set of information includes at least one target prediction result; The process of determining the second loss includes: Determining the loss corresponding to each of the target prediction results according to the label matching result, the second set of information, and the target label data; The second loss is determined according to an average value of the losses corresponding to the at least one target prediction result.

16. The method according to claim 1, characterized in that The target description information of the image to be processed is determined by using a pre-trained target tracking model; The training process of the target tracking model includes: Using at least one second image data and the target detection label of each second image data, training the initial model to obtain a model to be optimized; The target tracking model is obtained by training a decoding module in the model to be optimized using at least one image sequence and a target tracking label of each image sequence.

17. The method according to claim 1, characterized in that The at least one target includes one or more targets to be tracked; the tracking query representation data includes query representation data corresponding to each of the targets to be tracked under multiple historical images.

18. The method according to any one of claims 1 to 17, characterized in that: The image to be processed refers to an image extracted from an image sequence; the target description information includes at least one target prediction result; After obtaining the target description information of the image to be processed, the method further includes: Extracting target prediction results that meet preset reference conditions from the target description information; The image to be processed is updated using the next frame image in the image sequence, the tracking query characterization data is updated using the query characterization data in the target prediction result that meets the preset reference condition, and the step of performing feature extraction processing on the image to be processed is continued.

19. The method according to claim 18, characterized in that The image sequence is video data.

20. An image processing device, characterized in that: include: An extraction unit is used to perform feature extraction processing on the image to be processed to obtain a feature extraction result; An encoding unit, used for encoding the feature extraction result to obtain detection query representation data; A decoding unit, configured to perform decoding processing based on the detection query representation data and the tracking query representation data corresponding to the image to be processed, so as to obtain target description information of the image to be processed; The tracking query representation data is determined based on target description information of a plurality of historical images corresponding to the image to be processed; the target description information of the image to be processed is used to describe at least one target in the image to be processed.

21. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 19.

22. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 19.

Images