Image processing method and device, vehicle and storage medium

Abstract

This disclosure relates to an image processing method, apparatus, vehicle, and storage medium, and pertains to the field of image technology. The method includes: using a first query parameter of a first image as input parameters to a perception model to obtain a first predicted feature of an object in the first image; wherein the perception model is trained using a first query parameter sample of a first image sample and a second feature sample of an object in a second image sample, the first image sample being a previous frame image sample and the second image sample being a current frame image sample; determining the similarity between the first predicted feature and the second feature of an object in the second image; the first image being a previous frame image and the second image being the current frame image. Using the image processing method proposed in this disclosure, a more accurate similarity between the first predicted feature and the second feature in two adjacent frames can be obtained.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and more particularly to an image processing method, apparatus, vehicle, and storage medium. Background Technology

[0002] In autonomous driving scenarios, similarity can be calculated for objects in multiple frames of images, thereby identifying objects with high similarity in multiple frames as the same target.

[0003] In related technologies, when calculating the similarity between objects in multiple frames of images, some objects move, causing the same object to have positional differences in multiple frames. This positional difference affects the similarity calculation between objects, resulting in lower accuracy of the calculated similarity between objects. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus, vehicle, and storage medium.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0006] The first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image; wherein, the perception model is trained using the first query parameter sample of the first image sample and the second feature sample of the object in the second image sample, the first image sample is the previous frame image sample, and the second image sample is the current frame image sample;

[0007] Determine the similarity between the first predicted feature and a second feature of an object in the second image; the first image is the previous frame image, and the second image is the current frame image.

[0008] Optionally, determining the similarity between the first predicted feature and the second feature of an object in the second image includes:

[0009] The second query parameter of the second image is used as the input parameter of the perception model to obtain the second feature;

[0010] Determine the similarity between the first predicted feature and the second feature.

[0011] Optionally, determining the similarity between the first predicted feature and the second feature of an object in the second image includes:

[0012] Select the top N target features with the highest confidence from the first predictive feature and the second feature; N is greater than or equal to 1;

[0013] Determine the similarity between the target feature and the first predicted feature.

[0014] Optionally, determining the similarity between the target feature and the first predicted feature includes:

[0015] Filter out the first target feature that belongs to the second feature from the target features;

[0016] Determine the similarity between the first target feature and the first predicted feature.

[0017] Optionally, the method further includes:

[0018] From multiple first target features, select a first feature to be processed that has a similarity greater than a first preset value with the first predicted feature and has a maximum similarity with the first predicted feature;

[0019] Assign the identifier of the first predicted feature to the first feature to be processed.

[0020] Optionally, the method further includes:

[0021] From multiple first target features, a second feature to be processed is selected whose similarity to the first predicted feature is less than a second preset value.

[0022] Create an identifier for the second feature to be processed.

[0023] Optionally, the method further includes:

[0024] Select a second target feature that belongs to the first predicted feature from the target features;

[0025] Maintain the identifier of the second target feature.

[0026] Optionally, the step of using the first query parameters of the first image as input parameters of the perception model to obtain the first predicted features of objects in the first image includes:

[0027] For the first image in any view of a multi-view image, the first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image.

[0028] Determining the similarity between the first predicted feature and the second feature of an object in the second image includes:

[0029] Determine the similarity between the first predicted feature and a second feature of an object in a second image that is at the same viewpoint as the first image.

[0030] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0031] The prediction module is configured to use the first query parameters of the first image as input parameters of the perception model to obtain the first predicted features of the objects in the first image; wherein the perception model is trained using the first query parameter samples of the first image sample and the second feature samples of the objects in the second image sample, the first image sample being the previous frame image sample and the second image sample being the current frame image sample.

[0032] The similarity calculation module is configured to determine the similarity between the first predicted feature and a second feature of an object in the second image; the first image is the previous frame image, and the second image is the current frame image.

[0033] According to a third aspect of the present disclosure, a vehicle is provided, comprising:

[0034] processor;

[0035] Memory used to store processor-executable instructions;

[0036] The processor is configured as follows:

[0037] The steps of performing the image processing method provided in the first aspect of the embodiments of this disclosure.

[0038] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0039] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0040] By using the first query parameters of the first image in the previous frame to predict the first predicted feature of the first image in the current frame through a perceptual model, the similarity between the first predicted feature of the first image in the current frame and the second feature of the second image in the current frame is calculated. This ensures that the first predicted feature of the object in the first image and the second feature of the object in the second image are both under the observation view of the current frame, eliminating the influence of inter-frame changes of the same object in different frames on the similarity calculation. As a result, the similarity calculated based on the first predicted feature of the first image and the second feature of the second image in the current frame is more accurate.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0043] Figure 1 This is a flowchart illustrating the steps of an image processing method according to an exemplary embodiment.

[0044] Figure 2 This is a logical schematic diagram illustrating an image processing method according to an exemplary embodiment.

[0045] Figure 3 This is a schematic diagram illustrating two adjacent image samples according to an exemplary embodiment.

[0046] Figure 4 This is a flowchart illustrating the steps of an image processing method according to an exemplary embodiment.

[0047] Figure 5 This is a flowchart illustrating the steps of an image processing method according to an exemplary embodiment.

[0048] Figure 6 This is a flowchart illustrating the steps of an image processing method according to an exemplary embodiment.

[0049] Figure 7 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0050] Figure 8 This is a block diagram illustrating a vehicle according to an exemplary embodiment.

[0051] Figure 9 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment.

[0052] Figure 10 This is a block diagram illustrating a chip system according to an exemplary embodiment. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0054] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0055] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 1 As shown, it includes the following steps.

[0056] In step S10, the first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image.

[0057] The first image and the second image mentioned in subsequent step S20 are two adjacent frames. The first image is the previous frame of the current frame, and the second image is the current frame.

[0058] The first query parameter (query) of the first image is a query vector used to query the first predicted features of objects in the first image. It indicates the objects that may exist in the first image and is used to query the objects that exist at a specified location in the first image. The first image has multiple first query parameters, which respectively indicate multiple objects that may exist in the first image.

[0059] The first predicted feature of an object in the first image includes the features of at least one object in the first image predicted by the perceptual model under the current frame observation view. The first predicted feature indicates information such as the moving speed, size (e.g., length, width, and height of the object), orientation, position, and type of the object in the first image under the current frame observation view.

[0060] In this embodiment of the disclosure, the perception model (transformer) has two functions:

[0061] The first function is that the perceptual model can predict the second feature of the object corresponding to the second query parameter in the second image of the current frame based on the second query parameter. It can also predict the second feature sample of the object corresponding to the second query parameter sample in the second image based on the second query parameter sample of the second image sample of the current frame.

[0062] The second function is that the perception model can switch the observation perspective of the first image in the previous frame to the observation perspective of the current frame based on the first query parameter of the first image in the previous frame, so as to predict the first predicted feature of the object corresponding to the first query parameter in the first image of the current frame.

[0063] Of the two functions mentioned above, the perception model is first trained for the first function, and then trained for the second function.

[0064] For the second function of the perception model, the first query parameter sample of the first image sample can be used as the training sample, and the second feature sample of the object in the second image sample can be used as the label to train the perception model. This enables the perception model to predict the first predicted feature of the object corresponding to the first query parameter of the first image in the current frame based on the first query parameter of the first image in the previous frame observation view.

[0065] The first query parameter sample of the first image sample is a query vector used to query the first predicted feature sample of objects in the first image sample, which indicates the objects that may exist in the first image sample.

[0066] The first image sample and the second image sample are two adjacent frame image samples, with the first image sample being the previous frame image sample and the second image sample being the current frame image sample.

[0067] The first predicted feature sample in the first image sample includes the first predicted feature sample of at least one object in the first image sample. The first predicted feature sample indicates information such as the moving speed, size, orientation, position and type of the object in the first image sample.

[0068] The second feature sample in the second image sample includes a second feature sample of at least one object in the second image sample. The second feature sample indicates information such as the moving speed, size, orientation, position and type of the object in the second image sample.

[0069] During training of the perceptual model, the following steps can be iteratively executed: Input the first query parameter sample of the first image sample from the previous round into the perceptual model. The perceptual model predicts the first predicted feature sample from the previous round. Then, calculate the loss value between the first predicted feature sample and the second feature sample. Determine if the loss value meets the convergence condition. If the loss value does not meet the convergence condition, update the network parameters of the perceptual model based on the loss value. Then, input the first query parameter sample of the first image sample from the next round into the perceptual model. The perceptual model predicts the first predicted feature sample from the next round. Calculate the loss value between the first predicted feature sample and the second image sample. This process is iterated until the loss value meets the convergence condition. When the loss value meets the convergence condition, the perceptual model with the convergent loss value is used as the trained perceptual model.

[0070] When updating the network parameters of the perception model based on the loss value, the parameters such as the weights and activation function of the perception model can be updated.

[0071] The convergence condition for the loss value is one of the following: the loss value is less than the preset value, or the loss value cannot continue to decrease.

[0072] By using the first query parameter sample of the object in the first image sample as the training sample and the second feature sample of the second image sample as the label of the training sample to train the perception model, the perception model can continuously approximate the second feature sample of the second image sample in the current frame based on the first predicted feature sample of the object output by the first query parameter sample of the first image sample in the previous frame. This allows the perception model to learn the potential relationship between the first query parameter sample of the first image sample in the previous frame and the second feature sample of the object in the second image sample in the current frame. In other words, the perception model learns the change between the first predicted feature sample corresponding to the first query parameter sample of the first image sample in the previous frame and the second feature sample of the second image sample in the current frame. This change is the inter-frame change of the same object in the first image sample in the previous frame and the second image sample in the current frame. Only when the first query parameter of the first image in the previous frame is received later can the first predicted feature of the first image in the current frame be predicted, thereby eliminating the inter-frame change of the same object in the previous frame and the current frame.

[0073] For example, taking the characteristics of position and movement speed as an example, please refer to Figure 3 As shown, the perceptual model can learn the positional change between the pedestrian's position (x1, y1) in the first image sample of the previous frame and the same pedestrian's position (x2, y2) in the first image sample of the current frame. Similarly, the perceptual model can learn the velocity change between the pedestrian's velocity v1 in the first image sample of the previous frame and the same pedestrian's velocity v2 in the second image sample of the current frame. Therefore, when the first query parameter of the first image of the previous frame is received subsequently, the perceptual model can predict the pedestrian's position in the first image from the current frame's observation perspective, and when the pedestrian's velocity in the first image of the previous frame is received, the perceptual model can predict the pedestrian's velocity in the first image from the current frame's observation perspective.

[0074] In step S20, the similarity between the first predicted feature and the second feature of the object in the second image is determined.

[0075] The second image is the current frame image. The second feature in the second image includes the second feature of at least one object in the second image. The second feature indicates information such as the moving speed, size, orientation, position and type of the object in the second image.

[0076] The similarity between the first predicted feature and the second feature can be regarded as the similarity between the first predicted feature of the object in the current frame observation view predicted by the perceptual model and the second feature of the object in the current frame observation view. It uses the first query parameter of the first image in the previous frame observation view to predict the first predicted feature of the first image in the current frame observation view, and then calculates the similarity between the first predicted feature of the first image in the current frame observation view and the second feature of the second image in the current frame observation view. This eliminates the influence of the inter-frame variation of the same object in different frame images on the similarity calculation.

[0077] In related technologies, due to the movement of objects, the same object will exhibit inter-frame changes in adjacent frames. These inter-frame changes include changes in position, orientation, and velocity of the same object in adjacent frames. Similarity calculation is to determine whether the objects in adjacent frames are the same target object by calculating the similarity of position, orientation, velocity, etc. between objects in adjacent frames. It can be seen that the inter-frame changes of the object itself will affect the similarity calculation between objects, resulting in low accuracy of the calculated similarity.

[0078] For example, in scenes with dense pedestrians in an image, traditional similarity calculation schemes that do not consider inter-frame changes can easily misidentify pedestrian A in the previous frame and pedestrian B in the current frame as the same target. If the two pedestrians move at different speeds, with pedestrian A moving faster and pedestrian B moving slower, an abnormal movement speed phenomenon will occur, where pedestrians in the previous frame move faster and pedestrians in the current frame move slower.

[0079] For example, in scenes with dense pedestrians in an image, traditional similarity calculation schemes can easily misidentify pedestrian A in the previous frame and pedestrian A in the current frame as two different pedestrians, thus causing the same pedestrian to be mistakenly split into two pedestrians.

[0080] In the above examples, whether it is an abnormal pedestrian movement speed or a pedestrian splitting apart, it will affect the control decisions in autonomous driving, causing the vehicle to misjudge the pedestrian's state, thus causing the vehicle to make incorrect braking operations to avoid the pedestrian, which affects the driving experience; it may also cause the vehicle to make incorrect driving operations and collide with the pedestrian, which can easily cause a safety accident.

[0081] The above technical solution uses a perception model to predict the first predicted feature of the first image in the current frame using the first query parameters of the first image in the previous frame. Then, it calculates the similarity between the first predicted feature of the first image in the current frame and the second feature of the second image in the current frame from the observation perspective. This ensures that the first predicted feature of the object in the first image and the second feature of the object in the second image are both under the observation perspective of the current frame, eliminating the influence of inter-frame changes of the same object in the previous and current frames on the similarity calculation. As a result, the similarity calculated based on the first predicted feature of the first image and the second feature of the second image in the current frame is more accurate.

[0082] Firstly, more accurate similarity avoids issues like pedestrian splitting or abnormal pedestrian movement speed, providing accurate guidance for subsequent vehicle control strategies. This improves the driving experience while reducing the occurrence of accidents. Furthermore, by shifting the observation perspective of the first image from the previous frame to the current frame, the perception model eliminates the inter-frame variation between the two. This allows the first predicted features in the first image of the previous frame, which could not previously match the second features in the second image of the current frame, to match. For example, pedestrian A in the first image of the previous frame and pedestrian A in the second image of the current frame will be matched as the same object, rather than splitting into two pedestrians, thus improving the matching success rate.

[0083] Secondly, this disclosure first trains the perception model to obtain the second feature sample based on the second query parameter of the second image sample, and then trains the perception model to switch the observation view of the first image sample in the previous frame to the observation view of the current frame. During the process of training the perception model to switch the observation view of the first image sample in the previous frame to the observation view of the current frame, it is necessary to annotate the second feature samples of the objects in the second image sample. Since the perception model can already output the second feature samples of each object in the second image sample based on the second query parameter sample of the second image sample in the current frame, there is no need to additionally annotate the second feature samples of each object in the second image sample, thus reducing the annotation cost.

[0084] Thirdly, since the second query parameter sample of the second image sample itself needs to be input into the perception model to obtain the second feature sample in the second image sample, in the process of training the perception model to switch the observation view of the first image sample of the previous frame to the observation view of the current frame, there is no need to configure the second feature sample as a supervision signal for the perception model, thus saving the configuration overhead of the supervision signal.

[0085] Figure 4This is an exemplary solution involving step S20 as illustrated in an exemplary embodiment of this disclosure, which includes the following steps:

[0086] In step S21, the second query parameter of the second image is used as the input parameter of the perception model to obtain the second feature.

[0087] In addition to eliminating inter-frame variations of the same object between adjacent frames, the perceptual model can also use the second query parameter of the second image to obtain the second feature of the object indicated by the second query parameter in the second image.

[0088] The second query parameter of the second image is a query vector used to query the second features of objects in the second image. It indicates the objects that may exist in the second image and is used to query the objects that exist at a specified location in the first image. The second image has multiple second query parameters, each indicating a possible object in the second image.

[0089] The second feature of an object in the second image includes at least one second feature of an object in the second image, the second feature indicating information such as the object's speed of movement, size (e.g., length, width, and height), orientation, position, and type.

[0090] In step S22, the similarity between the first predicted feature and the second feature is determined.

[0091] Optionally, the first predicted feature includes the first velocity, first position, first orientation, and first type of the object in the first image, and the second feature includes the second velocity, second position, second orientation, and second type of the object in the second image. The velocity difference between the first and second velocities, the position difference between the first and second positions, the direction difference between the first and second orientations, and the difference between the first and second types (1 for the same type and 0 for different types, or a probability between 0 and 1) can be calculated. The velocity difference is multiplied by a first weight, the position difference by a second weight, the direction difference by a third weight, and the category difference by a fourth weight to obtain a target value. This target value indicates the similarity between the first predicted feature and the second feature. The smaller the target value, the closer the velocities, positions, orientations, and types of the objects in the first and second images are, and the greater the similarity between the first predicted feature and the second feature.

[0092] The first, second, third, and fourth weights can be pre-configured according to the actual situation, and will not be elaborated here.

[0093] Optionally, please refer to Figure 2As shown, the vehicle will collect multi-view images. For the first image under any view in the multi-view images, the first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image. Then, the similarity between the first predicted feature and the second feature of the object in the second image under the same view of the first image is determined. Then, the multi-view images are fused according to the similarity between each feature in two adjacent frames of the multi-view images.

[0094] During fusion, for two adjacent frames of images from a single viewpoint, the similarity between various features in the two adjacent frames can be used to identify two objects with high similarity. Then, the same object with high similarity in the two adjacent frames is assigned the same label, and different objects are assigned different labels to distinguish different objects. In this way, the label of each object in the image from different viewpoints is obtained. Then, the multi-view images are fused. During fusion, objects with the same label in the multi-view images are fused together. In this way, the multi-view images are fused into a single image.

[0095] Multi-view images are images taken from different locations on a vehicle by different sensors, such as panoramic sensors and surround-view sensors, at the same location from different perspectives. By fusing multi-view images, feature information of an object can be obtained from different perspectives, thus making the representation of the object more complete and accurate.

[0096] Through the above technical solution, the second feature of the object in the second image can be obtained by using the second query parameter of the second image through the perception model. That is, the second feature of the object in the current frame image can be obtained by using the second query parameter of the current frame image through the perception model, and then the subsequent similarity calculation can be performed. When the similarity between the object in the first image of the previous frame and the object in the second image of the current frame is large, the two objects with large similarity are identified as the same target object, thereby realizing the tracking and detection of the target object.

[0097] Figure 5 An exemplary embodiment of the present disclosure is shown, which illustrates that feature filtering can be performed on the first predicted feature and the second feature before determining the similarity between the first predicted feature and the second feature, and includes the following steps:

[0098] In step S30, the top N target features with the highest confidence are selected from the first predicted features and the second features.

[0099] The target features are the top N features with the highest confidence among the first and second predictive features, where N is greater than or equal to 1. The confidence of the first predictive feature indicates the credibility of the first predictive feature, and the confidence of the second feature indicates the credibility of the second feature.

[0100] For example, the first query parameter represents the object at position A in the first image. After inputting the first query parameter of the first image into the perception model, the perception model will predict that the probability of the object at position A being a pedestrian is 80% and the probability of it being a robot is 20%. Since the probability of the object at position A being a pedestrian is relatively high, the perception model will output that the object at position A is a pedestrian with a confidence level of 80%.

[0101] For example, the second query parameter represents the object at position B in the second image. After inputting the second query parameter of the second image into the perception model, the perception model will predict that the probability of the object at position B being a kitten is 90% and the probability of it being a puppy is 10%. Since the probability of the object at position B being a kitten is relatively high, the perception model will output that the object at position B in the second image is a kitten with a confidence level of 90%.

[0102] In some scenarios, after inputting the three first query parameters of the first image into the perception model, the confidence levels of the first predicted features of the three objects, pedestrian, cat, and dog, are 80%, 70%, and 50%, respectively. After inputting the three second query parameters of the second image into the perception model, the confidence levels of the second features of the three objects, trees, roads, and guardrails, are 10%, 20%, and 95%, respectively. Assuming N is 2, taking the selection of the first two target features from the first and second predicted features as an example, the features of pedestrian, cat, and guardrail can be selected as target features from the first and second predicted features of the six objects, namely pedestrian, cat, dog, trees, roads, and guardrails.

[0103] In step S40, the similarity between the target feature and the first predicted feature is determined.

[0104] Continuing with the example in step S30 above, after selecting the features of the three objects—pedestrian, kitten, and guardrail—as target features, the similarity between these three target features and the first predicted feature is calculated respectively.

[0105] Optionally, a first target feature belonging to the second feature is selected from the target features; the similarity between the first target feature and the first predicted feature is determined.

[0106] The objects in the second image of the current frame do not have labels, while the objects in the first image of the previous frame already have their own labels. In this context, we can first determine whether the target features contain the first target features that belong to the current second features. If they do, it means that these first target features may be features of newly added objects or features of objects that already existed in the first image of the previous frame. Therefore, we need to filter out the first target features that belong to the second features from the target features. The first target features that belong to the second features need to be included in the similarity calculation between the first predicted features of objects in the first image. We calculate the similarity between these first target features and the first predicted features, and based on the similarity between the first target features and the first predicted features, we determine whether the first target features are features of newly added objects or features of objects that already existed in the first image of the previous frame.

[0107] For example, a method for determining that a first target feature and a first predicted feature belong to the same object includes: selecting a first feature to be processed from a plurality of first target features that has a similarity greater than a first preset value with the first predicted feature and has a similarity with the first predicted feature that reaches the maximum similarity; and assigning an identifier of the first predicted feature to the first feature to be processed.

[0108] When the similarity between the first target feature and the first predicted feature is greater than the first preset value, and the similarity between the first target feature and the first predicted feature reaches the maximum similarity, it indicates that the similarity between the first target feature and the first predicted feature is relatively large. The first target feature is the first feature to be processed that belongs to the same target object as the first predicted feature. Therefore, the original identifier of the first predicted feature can be assigned to the first feature to be processed, so that the same object in the first image of the previous frame and the second image of the current frame can be assigned the same identifier, thereby realizing the recognition and detection of the same object in different images.

[0109] For example, taking a first preset value of 50% as an example, the identifier of pedestrian A in the first image of the previous frame output by the perception model is 1 from the observation perspective of the current frame. If the similarity between the first target features of objects A, B, and C in the second image of the current frame and the first predicted features of pedestrian A in the first image is calculated to be 40%, 60%, and 70%, respectively, since the similarity between object C and the first predicted features is greater than the first preset value of 50%, and the similarity is the largest among multiple similarities, the identifier 1 of pedestrian A in the first image of the previous frame will be assigned to object C. Thus, object C and pedestrian A are associated with the same identifier 1, representing that object C is also pedestrian A, thereby realizing the tracking and detection of pedestrian A in two adjacent frames.

[0110] For example, a method for determining that a first target feature and a first predicted feature belong to different objects includes: selecting a second feature to be processed from a plurality of first target features, wherein the similarity between the second feature and the first predicted feature is less than a second preset value; and creating an identifier for the second feature to be processed.

[0111] The second preset value is less than or equal to the first preset value.

[0112] When the similarity between the first target feature and the first predicted feature is less than the second preset value, it indicates that the similarity between the first target feature in the second image of the current frame and the first predicted feature in the first image of the previous frame is small. The first target feature is a second feature to be processed that belongs to a different object from the first predicted feature. The object indicated by the second feature to be processed is not the same target object as the object indicated by the first predicted feature in the first image. The object indicated by the second feature to be processed is a newly added object. Therefore, a new identifier can be created for the second feature to be processed, which also means that a new identifier can be created for the object indicated by the second feature to be processed.

[0113] For example, taking the second preset value as 50%, the identifier of pedestrian A in the first image of the previous frame output by the perception model is 1 from the observation perspective of the current frame. If the similarity between the first target features of objects A, B and C in the second image of the current frame and the first predicted features of pedestrian A in the first image is calculated to be 40%, 60% and 70% respectively, since the similarity between object A and the first predicted features is less than the second preset value of 50%, it means that object A and pedestrian A are not the same target object. Therefore, an additional identifier 2 can be added to object A to distinguish object A from pedestrian A.

[0114] Optionally, a second target feature belonging to the first predicted feature is selected from the target features; the identifier of the second target feature is retained.

[0115] First, it can be determined whether the target features contain second target features that belong to the first predicted features. If the second target features are present, it means that these second target features are features of objects that already exist in the first image of the previous frame. Since the second target features have already been assigned an identifier in the previous frame, the identifier of the second target features can be maintained, and features with the same identifier in the second image can be replaced with the second target features, so that the confidence of features represented by the same identifier in the second image is higher.

[0116] The following example illustrates the above scheme. When the similarity between the second feature in the second image of the current frame and the first predicted feature in the first image of the previous frame is high, it is considered that the second feature in the second image and the first predicted feature in the first image belong to the same object. Therefore, the identifier of the first predicted feature in the first image of the previous frame can also be assigned to the second feature in the second image of the current frame, so that the features indicated by the same identifier in adjacent frames are features of the same object, thus achieving tracking and recognition of the same object. When the similarity between the second feature in the second image of the current frame and the first predicted feature in the first image of the previous frame is low, it is considered that the second feature in the second image and the first predicted feature in the first image belong to different objects. That is, the second feature in the second image is a feature of a newly added object. Therefore, an additional identifier will be created for the second feature in the second image to distinguish it from other objects.

[0117] In this context, please refer to Figure 6 As shown, the first image of the previous frame contains objects 1, 2, and 3, with confidence levels of A, B, and C for objects 1, 2, and 3, respectively. After inputting the first query parameters of the three objects into the perceptual model, the first image of the previous frame is converted into the first image under the observation view of the current frame. The confidence levels of the three objects in the first image under the observation view of the current frame change, and the changed confidence levels are A', B', and C', respectively. The second image of the current frame also contains three objects: objects 1, 4, and 5, with confidence levels of A, E, and F, respectively. After inputting the second query parameters of the three objects into the perceptual model, the confidence levels of the three objects in the second image of the current frame change, and the changed confidence levels are A'', E', and F', respectively.

[0118] Suppose that the confidence level A' of object 1 and the confidence level E' of object 4 are the top two confidence levels among multiple objects in the first and second images. Since object 1 is the object 1 in the first image of the previous frame, the identifier of object 1 has already been marked in the first image of the previous frame. Therefore, the identifier of object 1 will be retained, and the features of object 1 with confidence level A” in the second image of the current frame will be replaced with the features of object 1 with higher confidence level A'.

[0119] Since the confidence level E' of object 4 is the confidence level of object 4 in the second image of the current frame, and object 4 is located in the second image of the current frame, the system does not know whether object 4 has appeared in the first image. Therefore, it will calculate the similarity between the features of object 4 and each feature in the first image under the observation view of the current frame, that is, calculate the similarity between the features of object 4 and object 1, and object 2 and object 3 respectively, to determine whether object 4 is an object that appeared in the first image or a newly added object.

[0120] If the similarity between object 4 and object 2 is high, the identifier of object 2 will be assigned to object 4, indicating that object 4 and object 2 are the same object; if the similarity between object 4 and any of the objects, such as object 1, object 2 and object 3, is low, a new identifier will be added to object 4 to indicate that object 4 is a new object, in order to distinguish it from other objects.

[0121] Through the above technical solution, firstly, the top N target features with the highest confidence can be selected from the first predicted features and the second features, and then the similarity between the target features and multiple first predicted features in the first image can be calculated, without having to include all the first predicted features and the second features in the similarity calculation, thereby reducing the amount of calculation for similarity calculation and improving the calculation speed.

[0122] Secondly, if there is a second target feature that belongs to the first predicted feature in the target feature, the identifier of the second target feature can be directly maintained, and the second feature of the same object in the second image can be replaced with the second target feature with higher confidence, without having the second target feature participate in the similarity calculation. On the one hand, this reduces the computational effort, and on the other hand, replacing the second feature of the same object in the second image with the second target feature with higher confidence can ensure the confidence of the object.

[0123] Thirdly, if a first target feature belonging to the second feature exists in the target feature, the similarity between the first target feature and the first predicted feature can be determined. If the similarity is large, the first target feature and the first predicted feature are considered to represent the same object, and the identifier of the first predicted feature can be assigned to the first target feature to represent the same object. If the similarity is small, the first target feature and the first predicted feature are considered to represent different objects, and an identifier can be added to the first target feature to distinguish it from other objects.

[0124] Understandably, please refer to Figure 2 As shown, after the identifiers of each object in the second image of the current frame are determined, before calculating the similarity between the third features of each object in the third image of the next frame and the second features of each object in the second image of the current frame, the second query parameters of each object in the second image of the current frame can be added to the queue of the first query parameters of the first image of the previous frame. That is, before calculating the similarity between the third features of each object in the third image of the next frame and the second features of each object in the second image of the current frame, the second query parameters of each object in the second image of the current frame are added to the historical query parameter queue to participate in the subsequent similarity calculation.

[0125] Figure 7 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 7 The image processing device 700 includes a prediction module 710 and a similarity calculation module 720.

[0126] The prediction module 710 is configured to use the first query parameters of the first image as input parameters of the perception model to obtain the first predicted features of the object in the first image; wherein the perception model is trained using the first query parameter samples of the first image sample and the second feature samples of the object in the second image sample, the first image sample being the previous frame image sample and the second image sample being the current frame image sample.

[0127] The similarity calculation module 720 is configured to determine the similarity between the first predicted feature and a second feature of an object in the second image; the first image is a previous frame image and the second image is a current frame image.

[0128] Optionally, the similarity calculation module 720 includes:

[0129] The prediction submodule is configured to use the second query parameter of the second image as the input parameter of the perception model to obtain the second feature;

[0130] The similarity calculation submodule is configured to determine the similarity between the first predicted feature and the second feature.

[0131] Optionally, the similarity calculation submodule includes:

[0132] The input submodule is configured to use the second query parameters of the second image as input parameters of the perception model to obtain the second feature;

[0133] The first similarity calculation submodule is configured to determine the similarity between the first predicted feature and the second feature.

[0134] Optionally, the similarity calculation submodule includes:

[0135] The first filtering submodule is configured to filter the top N target features with the highest confidence from the first predicted feature and the second feature; N is greater than or equal to 1;

[0136] The second similarity submodule is configured to determine the similarity between the target feature and the first predicted feature.

[0137] Optionally, the second similarity submodule includes:

[0138] The second filtering submodule is configured to filter out a first target feature belonging to the second feature from the target features;

[0139] The third similarity submodule is configured to determine the similarity between the first target feature and the first predicted feature.

[0140] Optionally, the image processing apparatus 700 includes:

[0141] The first filtering module is configured to filter out a first feature to be processed from a plurality of first target features, wherein the similarity between the first predicted feature and the first predicted feature is greater than a first preset value and the similarity between the first predicted feature and the first predicted feature reaches the maximum similarity.

[0142] The assignment module is configured to assign the identifier of the first predicted feature to the first feature to be processed.

[0143] Optionally, the image processing apparatus 700 includes:

[0144] The second filtering module is configured to filter out a second feature to be processed from a plurality of the first target features, wherein the similarity between the second feature and the first predicted feature is less than a second preset value.

[0145] The module is configured to create an identifier for the second feature to be processed.

[0146] Optionally, the image processing apparatus 700 includes:

[0147] The third filtering module is configured to filter out second target features belonging to the first predicted feature from the target features;

[0148] The retention module is configured to retain the identifier of the second target feature.

[0149] Optionally, the prediction module 710 is further configured to take the first query parameter of the first image as the input parameter of the perception model for the first image under any view in the multi-view image, and obtain the first predicted feature of the object in the first image.

[0150] The similarity calculation module 720 is also configured to determine the similarity between the first predicted feature and a second feature of an object in a second image that is at the same viewpoint as the first image.

[0151] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0152] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the image processing method provided in this disclosure.

[0153] Figure 8This is a block diagram illustrating a vehicle 800 according to an exemplary embodiment. For example, vehicle 800 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 800 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0154] Reference Figure 8 The vehicle 800 may include various subsystems, such as an infotainment system 810, a perception system 820, a decision control system 830, a drive system 840, and a computing platform 850. The vehicle 800 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 800 can be interconnected via wired or wireless means.

[0155] In some embodiments, the infotainment system 810 may include a communication system, an entertainment system, and a navigation system, etc.

[0156] The perception system 820 may include several sensors for sensing information about the environment surrounding the vehicle 800. For example, the perception system 820 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0157] The decision control system 830 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0158] The drive system 840 may include components that provide powered motion to the vehicle 800. In one embodiment, the drive system 840 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0159] Some or all of the functions of the vehicle 800 are controlled by a computing platform 850. The computing platform 850 may include at least one processor 851 and a memory 852, the processor 851 being able to execute instructions 853 stored in the memory 852.

[0160] The processor 851 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0161] The memory 852 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0162] In addition to instruction set 853, memory 852 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 852 can be used by computing platform 850.

[0163] In this embodiment of the disclosure, processor 851 may execute instructions 853 to complete all or part of the steps of the above-described image processing method.

[0164] Figure 9 This is a block diagram illustrating an image processing apparatus 900 according to an exemplary embodiment. For example, apparatus 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0165] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output interface 912, a sensor component 914, and a communication component 916.

[0166] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the image processing method described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0167] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0168] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 900.

[0169] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0170] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0171] Input / output interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0172] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in position of device 900 or a component of device 900, the presence or absence of user contact with device 900, orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0173] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0174] In an exemplary embodiment, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.

[0175] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by the processor 920 of the device 900 to complete the image processing method described above. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0176] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image processing method described above when executed by the programmable device.

[0177] Some embodiments of this disclosure also provide a chip system, such as Figure 10 As shown, the chip system includes at least one processor 1001 and at least one interface circuit 1002. The processor 1001 and the interface circuit 1002 are interconnected via lines. For example, the interface circuit 1002 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1002 can be used to send signals to other devices (e.g., the processor 1001). Exemplarily, the interface circuit 1002 can read instructions stored in memory and send those instructions to the processor 1001. When the instructions are executed by the processor 1001, the image processing device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and some embodiments of this disclosure do not specifically limit this.

[0178] In some embodiments of this disclosure, the interface circuit 1002 can acquire data, program instructions, and / or information from the internal storage area of ​​the chip system; it can also acquire data, program instructions, and / or information from outside the chip system.

[0179] Optionally, the chip system may also include a memory for storing necessary computer programs and data.

[0180] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: The first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image; wherein, the perception model is trained using the first query parameter sample of the first image sample and the second feature sample of the object in the second image sample, the first image sample is the previous frame image sample, and the second image sample is the current frame image sample; Determine the similarity between the first predicted feature and a second feature of an object in the second image; the first image is the previous frame image, and the second image is the current frame image.

2. The method according to claim 1, characterized in that, Determining the similarity between the first predicted feature and the second feature of an object in the second image includes: The second query parameter of the second image is used as the input parameter of the perception model to obtain the second feature; Determine the similarity between the first predicted feature and the second feature.

3. The method according to claim 1, characterized in that, Determining the similarity between the first predicted feature and the second feature of an object in the second image includes: Select the top N target features with the highest confidence from the first predictive feature and the second feature; N is greater than or equal to 1; Determine the similarity between the target feature and the first predicted feature.

4. The method according to claim 3, characterized in that, Determining the similarity between the target feature and the first predicted feature includes: Filter out the first target feature that belongs to the second feature from the target features; Determine the similarity between the first target feature and the first predicted feature.

5. The method according to claim 4, characterized in that, The method further includes: From multiple first target features, select a first feature to be processed that has a similarity greater than a first preset value with the first predicted feature and has a maximum similarity with the first predicted feature; Assign the identifier of the first predicted feature to the first feature to be processed.

6. The method according to claim 4, characterized in that, The method further includes: From multiple first target features, a second feature to be processed is selected whose similarity to the first predicted feature is less than a second preset value. Create an identifier for the second feature to be processed.

7. The method according to claim 3, characterized in that, The method further includes: Select a second target feature that belongs to the first predicted feature from the target features; Maintain the identifier of the second target feature.

8. The method according to claim 1, characterized in that, The step of using the first query parameters of the first image as input parameters of the perception model to obtain the first predicted features of objects in the first image includes: For the first image in any view of a multi-view image, the first query parameter of the first image is used as the input parameter of the perception model to obtain the first predicted feature of the object in the first image. Determining the similarity between the first predicted feature and the second feature of an object in the second image includes: Determine the similarity between the first predicted feature and a second feature of an object in a second image that is at the same viewpoint as the first image.

9. An image processing apparatus, characterized in that, include: The prediction module is configured to use the first query parameters of the first image as input parameters of the perception model to obtain the first predicted features of the objects in the first image; wherein the perception model is trained using the first query parameter samples of the first image sample and the second feature samples of the objects in the second image sample, the first image sample being the previous frame image sample and the second image sample being the current frame image sample. The similarity calculation module is configured to determine the similarity between the first predicted feature and a second feature of an object in the second image; the first image is the previous frame image, and the second image is the current frame image.

10. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: Perform the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images