Target determination method, electronic equipment and computer readable medium

Abstract

The invention provides a target determination method. The method comprises the following steps: acquiring first prediction position information and at least one piece of detection position information of a target object; obtaining first feature information corresponding to the target object according to the first prediction position information, and obtaining at least one piece of second feature information corresponding to the target object according to the at least one piece of first detection position information; wherein the first feature information and the second feature information comprise thermal imaging features of the target object; and determining target position information of the target object according to the first feature information and the at least one piece of second feature information. The invention further provides electronic equipment and a computer readable medium.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a target determination method, an electronic device, and a computer-readable medium. Background Art

[0002] Currently, when tracking and detecting a target object traveling on the road, a camera is usually used to collect multiple consecutive visible light images of the target object, and the position of the target object in the multiple visible light images is detected and classified to achieve continuous tracking and detection of the target object. However, when performing position detection on the target object based on the visible light image, there may be a problem of being unable to accurately determine the position of the target object. Summary of the invention

[0003] The present disclosure provides a target determination method, an electronic device, and a computer-readable medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a target determination method, including:

[0005] Acquire first predicted position information and at least one detected position information of a target object;

[0006] Obtaining first feature information corresponding to the target object according to the first predicted position information, and obtaining at least one second feature information corresponding to the target object according to the at least one first detected position information; wherein the first feature information and the second feature information include thermal imaging features of the target object;

[0007] The target position information of the target object is determined according to the first feature information and the at least one second feature information.

[0008] In a second aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a memory on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the target determination method described in the first aspect of the embodiment of the present disclosure.

[0009] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the target determination method described in the first aspect of the embodiment of the present disclosure is implemented.

[0010] In the target determination method provided by the embodiment of the present disclosure, when determining the position of the target object to track and detect the target object, first predicted position information and at least one first detection position information of the target object are obtained; then, first feature information corresponding to the target object and containing the thermal imaging feature of the target object is extracted based on the first predicted position information, and at least one second feature information corresponding to the target object and also containing the thermal imaging feature of the target object is extracted based on the at least one first detection position information. Since the thermal imaging feature is not affected by ambient light and will not be affected by other objects when the target object is close to other objects, the target position information of the target object can be accurately determined based on the first feature information and the at least one second feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flow chart of a target determination method provided for an embodiment of the present disclosure;

[0012] Figure 2 A first flow chart for obtaining feature information provided by an embodiment of the present disclosure;

[0013] Figure 3 A second flow chart for obtaining feature information provided by an embodiment of the present disclosure;

[0014] Figure 4 A schematic diagram of obtaining a coordinate transformation matrix provided in an embodiment of the present disclosure;

[0015] Figure 5 A flowchart for obtaining detection position information provided by an embodiment of the present disclosure;

[0016] Figure 6 A block diagram of an electronic device provided in an embodiment of the present disclosure;

[0017] Figure 7 A block diagram of a computer-readable medium provided for an embodiment of the present disclosure;

[0018] Figure 8 A schematic diagram of target determination processing provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the target determination method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0020] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0021] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0022] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0024] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0025] Currently, when tracking and detecting a target object traveling on the road, such as a vehicle, a camera can generally be used to capture multiple consecutive frames of visible light images of the target object, and the target object can be continuously tracked and detected by determining the position of the target object in the multiple consecutive frames of visible light images.

[0026] However, this method of continuously tracking and detecting the target object based on the visible light image may not be able to detect the detection frame of the target object in the visible light image when the ambient light is weak or the target object is close to other objects. That is, the detection frame may be lost and the position of the target object cannot be accurately determined, which leads to the problem of not being able to continuously track and detect the target object. The detection frame, also known as the bounding box, can specifically be a rectangular box to indicate the position and size of the target object. In the field of computer vision, the goal of bounding box regression is to predict the most suitable bounding box based on the target object in the image or video, so as to accurately surround the target object.

[0027] In view of this, in an embodiment of the present disclosure, when it is necessary to determine the position of a target object in order to track and detect the target object, first predicted position information and at least one first detection position information of the target object are obtained; thereafter, first feature information corresponding to the target object and containing the thermal imaging feature of the target object is extracted based on the predicted position information, and at least one second feature information corresponding to the target object and also containing the thermal imaging feature of the target object is extracted based on the at least one first detection position information. Since the thermal imaging feature is not affected by ambient light and will not be affected by other objects when the target object is close to other objects, the target position information of the target object can be accurately determined based on the first feature information and the at least one second feature information.

[0028] Figure 1 A flowchart of a target determination method provided in an embodiment of the present disclosure.

[0029] First, refer to Figure 1 The embodiment of the present disclosure provides a method for determining a target, which can be executed by an electronic device such as a terminal device or a server. The terminal device can be a vehicle-mounted device, a user equipment (UE, User Equipment), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA, Personal Digital Assistant), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory. Alternatively, the method can be executed by a server.

[0030] like Figure 1 As shown, the method may include the following steps S1-S3, which are described in detail below.

[0031] S1. Obtain first predicted position information and at least one first detected position information of a target object.

[0032] The target object may be any one or more objects that need to be tracked and detected. For example, the target object may be one or more vehicles traveling on the road.

[0033] In some embodiments, when the method is applied to the field of single object tracking (SOT), the target object may be, for example, a single vehicle traveling on the road; when the method is applied to the field of multiple object tracking (MOT), the target object may be, for example, multiple vehicles traveling on the road, without special limitation here; of course, the target object may also be an object in other scenarios, such as individuals, ships, and other objects in scenarios such as security, search and rescue, and the like.

[0034] The first predicted position information can be used to represent the predicted position of the target object at the current moment. In the case where the predicted position of the target object is represented by a prediction box, the first predicted position information can be the coordinate information corresponding to the prediction box, for example, it can be the information of the coordinates of the four vertices of the prediction box, or it can also be the information of the coordinates of the four vertices and the center point of the prediction box, which is not specifically limited here; wherein, the prediction box can be a bounding box used to represent the predicted position of the target object in the collected image or video frame containing the target object, and the image or video frame can be at least one of the images such as a visible light image, a thermal imaging image, a radar, and a laser point cloud image containing the target object, which is not specifically limited here.

[0035] In some embodiments, the predicted position may be the position of the target object at the current moment, i.e., the current frame, predicted based on the position and speed of the target object at the previous moment, i.e., the previous frame. The predicted position may be, for example, the possible position of the target object at the current moment predicted based on a Kalman filter.

[0036] The first detection position information may be information used to indicate the position of the target object obtained by acquiring an image or video frame containing the target object and performing object detection (Object Detection) processing on the image or video frame. In some embodiments, the first detection position information may be information of a detection frame corresponding to the target object in the image. For example, after performing object detection processing on the image, N detection frames are obtained to indicate the position of the target object, where N≥1. Then, based on the N detection frames, N detection position information may be obtained to indicate the detection position of the target object, i.e., the observation position. The detection position information may be information of the coordinates of the four vertices of the corresponding detection frame, or may be information of the coordinates of the four vertices and the center point of the corresponding detection frame, which is not specifically limited here.

[0037] S2. Obtaining first feature information corresponding to the target object according to the first predicted position information, and obtaining at least one second feature information corresponding to the target object according to at least one first detected position information; wherein the first feature information and the second feature information include thermal imaging features of the target object;

[0038] S3. Determine target location information of the target object based on the first feature information and at least one second feature information.

[0039] In the field of computer vision, in the related art, when it is necessary to continuously track and detect a target object to determine the position of the target object, the method usually adopted is: 1. Initialize a position tracker, such as initializing a Kalman filter corresponding to the target object, and the Kalman filter may include motion state parameters such as the initial position and initial speed of the target object at the initial moment; 2. When continuously tracking and detecting the target object, on the one hand, the position tracker predicts the predicted position P_P2 of the target object at the current moment T2 based on the position D_P1 and speed V1 of the target object at the previous moment T1; on the other hand, based on an image sensor, such as a camera, a visible light image of the target object at the current moment is collected, and at least one detection position O_P2 of the target object is detected based on the visible light image, wherein the at least one detection position may be the position of at least one detection frame corresponding to the target object in the visible light image, or may also be other positions in the visible light image that are related to the target object. Detection frame of objects with similar size or close distance to the target object; 3. In order to determine the accurate position D_P2 of the target object at the current moment, extract the visible light feature 1 of the target object in the visible light image based on the predicted position P_P2, and extract at least one visible light feature 2 of the target object in the visible light image based on the at least one detection position O_P2, wherein the visible light feature may include color features, contour features and other features corresponding to the target object in the visible light image; 4. By matching the visible light feature 1 with the at least one visible light feature 2 respectively, select the detection position corresponding to the visible light feature 2 with the highest matching degree as the target position D_P2 of the target object at the current moment T2; 5. Update the motion state parameters such as the position and speed of the position tracker according to the target position D_P2, so as to predict the predicted position of the target object at the next moment T3 based on the updated motion state parameters, thereby realizing continuous tracking and detection of the target object. It should be noted that in the field of multi-target tracking, since there may be multiple target objects that need to be continuously tracked and detected, in actual implementation, the number of predicted positions may also be multiple, and each predicted position corresponds to an object.

[0040] According to the above description, it can be known that in the related art, when continuously tracking a target object to determine the position of the target object, it is usually necessary to rely on observation values ​​obtained based on visible light images, that is, detection position information to determine the target position of the target object. However, when detecting the detection position of the target object based on the visible light image, in complex ambient light such as strong light, backlight, weak light, or when the target object is close to other objects, the detection frame of the target object may be lost, resulting in the inability to obtain the detection position of the target object, and then the inability to accurately determine the target position of the target object.

[0041] Of course, the above is to determine the target position of the target object based on feature matching after obtaining the predicted position and detection position of the target object. In actual implementation, there are also methods for determining the target object based on overlap (IOU, Intersection Over Union), based on target edges, and based on re-identification (ReID, Re-IDentification). However, when the number of target objects is large, the IOU-based method is usually more prone to frame loss, that is, the problem of inability to obtain the detection position; the target edge-based method has high requirements on the clarity and background of the visible light image and is more computationally intensive. It is easy to mismatch after the target posture is adjusted, especially in low-light environments. Matching errors are prone to occur; and the ReID method has a problem of poor universality due to its high computing power requirements.

[0042] To this end, in the embodiments of the present disclosure, taking into account the advantages that thermal imaging features are not affected by ambient light and other objects, when determining the target position of the target object, the embodiments of the present disclosure obtain first feature information and at least one second feature information including the thermal imaging feature of the target object based on the first predicted position information and at least one first detection position information corresponding to the target object, and determine the position of the target object based on the feature information including the thermal imaging feature, thereby accurately obtaining the accuracy of the target object and improving the accuracy and robustness of tracking and detection of the target object.

[0043] In some embodiments, see Figure 2 In the above step S2, obtaining first feature information corresponding to the target object according to the first predicted position information, and obtaining at least one second feature information corresponding to the target object according to at least one first detected position information, include:

[0044] S21, extracting a first thermal imaging feature from a position corresponding to the first predicted position information in the first image, and obtaining first feature information according to the first thermal imaging feature, wherein the first image is a thermal imaging image including the target object;

[0045] S22. Extract at least one second thermal imaging feature from a position corresponding to at least one first detection position information in the first image, and obtain at least one second feature information according to the at least one second thermal imaging feature.

[0046] The first image may be an image generated after a thermal imaging device, such as a thermal imager, collects thermal imaging data corresponding to the target object.

[0047] In the embodiment of the present disclosure, the first predicted position information can be used to indicate the predicted position of the target object in the first image; the first detected position information can be used to indicate the detected position of the target object in the first image.

[0048] For example, an infrared thermal imager or an infrared camera can be used to collect infrared radiation signals in the area where the target object is located. After obtaining the infrared radiation signal, the signal is first subjected to noise reduction processing, for example, the signal is amplified to increase the signal-to-noise ratio, and then filtered to remove high-frequency and low-frequency noise, and then corrected to ensure the accuracy of temperature measurement; for the infrared radiation signal after noise reduction processing, it is first converted into a numerical format, and then, through false color mapping, different temperature ranges are mapped to different colors or gray levels to intuitively represent the temperature distribution. For example, a higher temperature area can be mapped to a warm color (such as red or yellow), and a lower temperature area can be mapped to a cold color (such as blue or green); then, the mapped data can be used to generate a visual thermal imaging image. Of course, this is just a simple explanation of how to generate a thermal imaging image, and its detailed implementation process will not be repeated here.

[0049] In the embodiments of the present disclosure, the thermal imaging feature may include at least one of the temperature feature, color feature, and contour feature corresponding to the target object in the thermal imaging image, wherein the temperature feature may be a feature for representing the temperature distribution on the surface of the target object; the color feature may be a feature of the thermal imaging feature for representing colors of different temperature levels; and the contour feature may be a feature for representing the material, texture, and shape of the target object.

[0050] That is, in this embodiment, it is possible to directly rely on acquiring a thermal imaging image corresponding to the target object, that is, the first image, to achieve continuous tracking and detection of the target object. After acquiring the first image of the target object at the current moment, the first thermal imaging feature at the position corresponding to the first predicted position information in the first image can be extracted, and the second thermal imaging feature at the position corresponding to the first detected position information in the first image can be extracted. Since thermal imaging features are usually not affected by ambient light or other objects, the target position information of the target object can be determined based on the matching result by matching the first thermal imaging feature and the at least one second thermal imaging feature.

[0051] In some embodiments, in the above step S3, determining the target position information of the target object based on the first feature information and the at least one second feature information includes: matching the first thermal imaging feature with the at least one second thermal imaging feature to obtain a first matching result; and determining the target position information based on the first matching result.

[0052] The first matching result may include at least one first matching degree, each first matching degree and a second thermal imaging feature object, and the first matching degree may be used to indicate the degree of matching between the first thermal imaging feature and the corresponding second thermal imaging feature.

[0053] In some embodiments, the first matching degree can be obtained by calculating the distance between the first thermal imaging feature and the corresponding second thermal imaging feature, for example, Euclidean distance or cosine similarity, etc., which is not particularly limited here.

[0054] It can be understood that the first thermal imaging feature and the second thermal imaging feature can be at least one of the temperature feature, color feature and contour feature corresponding to the target object obtained based on the first image. In the process of feature matching, the corresponding features can be matched, for example, the temperature feature in the first thermal imaging feature is matched with the temperature feature in the second thermal imaging feature.

[0055] In some embodiments, determining the target position information according to the first matching result may be: obtaining a first matching degree satisfying a preset condition as a target matching degree from at least one first matching degree; and determining the target position information according to the first detection position information corresponding to the target matching degree.

[0056] The acquiring, from the at least one first matching degree, a first matching degree satisfying a first preset condition as the target matching degree may be acquiring the first matching degree with the highest value as the target matching degree.

[0057] That is, the first detection position information with the highest matching degree between the corresponding second thermal imaging feature and the first thermal imaging feature may be used as the target position information.

[0058] Since thermal imaging features are not affected by factors such as lighting conditions, background interference, and target occlusion, determining the target position information of the target object based on thermal imaging features to continuously track and detect the target object can improve the accuracy and reliability of the obtained target position information, thereby improving the reliability and accuracy of target tracking.

[0059] In some embodiments, the first predicted position information is obtained based on the second predicted position information, the second position information is used to indicate the predicted position of the target object in the second image, the at least one first detected position information includes position information obtained based on the at least one second detected position information, the second detected position information is used to indicate the detected position of the target object in the second image, and the second image includes other images except the thermal imaging image; see Figure 3 In this embodiment, in the above step S2, obtaining first feature information corresponding to the target object according to the first predicted position information, and obtaining at least one second feature information corresponding to the target object according to at least one first detected position information include:

[0060] S23, obtaining a coordinate conversion matrix between a first coordinate space corresponding to the first image and a second coordinate space corresponding to the second image;

[0061] S24, performing position conversion processing on the second predicted position information and the at least one second detected position information respectively according to the coordinate conversion matrix to obtain the first predicted position information corresponding to the second predicted position information and the at least one first detected position information corresponding to the at least one second detected position information;

[0062] S25. Extract a first thermal imaging feature from a position represented by the first predicted position information in the first image, and extract at least one second thermal imaging feature from a position represented by at least one first detected position information in the first image.

[0063] Specifically, in the embodiments of the present disclosure, continuous tracking and detection of the target object can be performed based on images of multiple modalities to accurately determine the position of the target object. Specifically, the first image can be a thermal imaging image, and the second image can be an image other than the thermal imaging image, for example, a visible light image. Of course, in actual implementation, the second image can also be an image such as a radar or laser point cloud image. For ease of explanation, the following example takes the first image as a thermal imaging image and the second image as a visible light image.

[0064] Since the thermal imaging device used to capture the first image and the image sensor used to capture the second image, such as the camera, are located at different positions, and the internal parameters of the different devices are also different, the first coordinate space corresponding to the first image and the second coordinate space corresponding to the second image are often different. For example, the pixel point at the coordinate position (x1, y1) of the first image may be located at the position represented by the coordinates (x2, y2) in the second image, where x1≠x2, y1≠y2; for this reason, in the embodiment of the present disclosure, when extracting the thermal imaging feature at the corresponding position from the first image, it is necessary to extract the thermal imaging feature based on the position information in the first coordinate space corresponding to the first image. For example, the thermal imaging feature corresponding to the target image can be extracted based on the first predicted position information in the first coordinate space of the first image obtained based on the second predicted position information and at least one first detection position information.

[0065] The following describes how to obtain the coordinate transformation matrix. Figure 4 Taking the first image acquired by the thermal imager 401 and the second image acquired by the camera 402 as an example, it is explained how to calibrate the internal and external parameters of the thermal imager 401 and the camera 402 to align the first image and the second image acquired by the two, establish the spatial relationship between the first coordinate space corresponding to the first image and the second coordinate space corresponding to the second image, that is, obtain the coordinate transformation matrix between the two.

[0066] First, the internal parameters of the thermal imager 401 and the camera 402 are calibrated respectively, which may include calibrating the focal length, distortion, sensor size and other internal parameters of the two to ensure that the images captured by the two have accurate geometric features. In actual implementation, when performing the internal parameter calibration process, any calibration method can be used to calibrate the internal parameters of the two. For example, the Zhang Zhengyou calibration method can be used to calibrate the internal parameters of the two. Figure 4As shown, external parameters such as the position and orientation of the thermal imager 401 and the camera 402 can be calibrated by placing calibration objects at known positions in the scene. For example, after fixing the thermal imager 401 and the camera 402, four points A, B, C, and D can be marked respectively for characteristic positions in the same scene picture. Then, the thermal imager 401 is used to capture the scene picture, and the coordinates of the four points A2, B2, C2, and D2 corresponding to the four points in the picture captured by the thermal imager 401 are obtained. Also, the camera 402 is used to capture the scene picture, and the coordinates of the four points A1, B1, C1, and D1 corresponding to the four points in the picture captured by the camera are obtained. Then, the coordinate conversion matrix between the first coordinate space corresponding to the first image captured by the thermal imager 401 and the second coordinate space corresponding to the second image captured by the camera 402 can be obtained through the coordinates of the four points A2, B2, C2, and D2 and the coordinates of the four points A1, B1, C1, and D1. For example, the coordinate transformation matrix perspective_matix can be obtained through the opencv tool library's library function getPerspectiveTransform(src_points,dst_points), where src_points can be [A1, B1, C1, D1] and dst_points can be [A2, B2, C2, D2]. Finally, when the first image needs to be aligned with the second image, the opencv tool library's library function perspectiveTransform(targetpoints,perspective_matix) can be used. , convert the target coordinate points targetpoints in the first image or the second image to the other image. For example, when src_points is [A1, B1, C1, D1] and dst_points is [A2, B2, C2, D2], it can be seen that the coordinate transformation matrix obtained here is used to transform the position information in the second image into the first image. If the target coordinate point to be converted is a coordinate point in the second image, the target coordinate point can be directly transformed into the first image according to the obtained coordinate transformation matrix. If the target coordinate point is a coordinate point in the first image, the target coordinate point can be transformed into the second image using the inverse matrix of the coordinate transformation matrix.

[0067] The above is only an example of how to obtain the coordinate conversion matrix between the first image and the second image. In actual implementation, the coordinate conversion matrix can also be obtained by other methods, which are not specifically limited here.

[0068] After obtaining the coordinate conversion matrix, if the initially obtained second predicted position information and second detected position information are coordinate points in the second coordinate space corresponding to the second image, the two position information can be converted through the coordinate conversion matrix to obtain the first predicted position information and the first detected position information in the first coordinate space corresponding to the first image, so as to accurately extract the first thermal imaging feature and the second thermal imaging feature from the corresponding positions of the first image, avoiding the incorrect determination of the position of the target object due to an incorrect feature extraction range.

[0069] In some embodiments, the second image may be a visible light image. In this embodiment, the first feature information may also include a first visible light image feature, which is a feature extracted from the position represented by the second predicted position information in the second image; the second feature information also includes a second visible light image feature, which is a feature extracted from the position represented by the second detected position information in the second image; in this embodiment, in the above step S3, determining the target position information of the target object based on the first feature information and at least one second feature information includes: matching the first thermal imaging feature with at least one second thermal imaging feature to obtain a first matching result; matching the first visible light image feature with at least one second visible light image feature to obtain a second matching result; determining the target position information of the target object based on the first matching result and the second matching result.

[0070] It should be noted that the visible light image features described in the embodiments of the present disclosure may be image features corresponding to the target object extracted from the second image, for example, may be traditional color, contour and other features corresponding to the target object in the second image.

[0071] That is, in the embodiments of the present disclosure, on the one hand, when determining the position of the target object, a plurality of second matching results are obtained by using a matching method based on features extracted from a visible light image in the related technology. At the same time, a plurality of first matching results are obtained by introducing an additional thermal imaging feature matching method, so as to supplement the plurality of second matching results obtained in the related technology based on the plurality of first matching results, thereby improving the matching accuracy.

[0072] In some embodiments, determining the target location information of the target object based on the first matching result and the second matching result can be as follows: weighting multiple first matching degrees in the first matching result and multiple second matching degrees in the second matching result to obtain multiple third matching degrees; selecting a third matching degree that meets a second preset condition from the multiple third matching degrees as the target matching degree; determining the target location information based on the detection location information corresponding to the target matching degree; wherein the second preset condition can, for example, be the third matching degree with the largest value among the multiple third matching degrees.

[0073] It can be seen that based on the method provided by the embodiment of the present disclosure, by collecting images of multiple modalities corresponding to the target object, namely the first image and the second image, in the process of tracking and detecting the target object, the thermal imaging features extracted from the first image are used to assist in determining the best match, which can cope with challenges such as different lighting conditions, background interference and target occlusion, and accurately determine the target position information of the target object, thereby improving the reliability and accuracy of target tracking.

[0074] In some embodiments, see Figure 5 In the above step S1, the step of obtaining at least one detection position information of the target object includes:

[0075] S11, acquiring a first image and a second image corresponding to the target object, wherein the first image includes a thermal imaging image, and the second image includes an image other than the thermal imaging image;

[0076] S12, obtaining at least one first position information corresponding to the target object in the first image, and obtaining at least one second position information corresponding to the target object in the second image;

[0077] S13. Obtain at least one first detection position information according to at least one first position information and at least one second position information.

[0078] The first position information is information used to represent the position of the target object in the first image. The first position information may be, for example, the coordinates of at least one detection box corresponding to the target object in the first image. Correspondingly, the second position information may be information used to represent the position of the target object in the second image. The second position information may be, for example, the coordinates of at least one detection box corresponding to the target object in the second image.

[0079] That is, in some embodiments, in step S12, the obtaining of at least one first position information corresponding to the target object in the first image, and the obtaining of at least one second position information corresponding to the target object in the second image, include: performing detection processing on the first image to obtain at least one first detection frame corresponding to the target object, and obtaining at least one first position information based on the at least one first detection frame; performing detection processing on the second image to obtain at least one second detection frame corresponding to the target object, and obtaining at least one second position information based on the at least one second detection frame.

[0080] In actual implementation, the detection processing of the first image and the detection processing of the second image may be inputting the first image or the second image into a pre-trained target detection model for target detection processing to obtain the at least one first detection frame and the at least one second detection frame, wherein the network structure of the target detection model may be a convolutional neural network model CNN, YOLO or other network structure, which is not specifically limited here.

[0081] In addition, the target detection model can be used to perform target detection on images of both the first image and the second image at the same time, and output the detection frames of the target objects in the images of the two modalities; or, two models can be trained separately to perform object detection on images of the two modalities respectively. The specific implementation method is not particularly limited here.

[0082] In some embodiments, in step S13, obtaining at least one detection position information based on at least one first position information and at least one second position information includes: matching at least one first position information and at least one second position information to obtain at least one first detection position information.

[0083] Among them, the matching processing of at least one first position information and at least one second position information to obtain at least one first detection position information includes: converting at least one first position information and at least one second position information into the same coordinate space for matching and deduplication processing to obtain at least one first detection position information.

[0084] That is, since the first coordinate space corresponding to the first image and the second coordinate space corresponding to the second image are often different, in the process of matching the first position information and the second position information, they need to be spatially aligned, that is, they need to be converted to the same coordinate space for matching processing to obtain at least one first detection position information corresponding to the target object.

[0085] In some embodiments, the first position information is in a first coordinate space corresponding to the first image, and the second position information is in a second coordinate space corresponding to the second image; the converting of at least one first position information and at least one second position information into the same coordinate space for matching processing to obtain at least one first detection position information includes: obtaining a coordinate conversion matrix corresponding to the first coordinate space and the second coordinate space; according to the coordinate conversion matrix, converting at least one second position information into the first coordinate space for matching and deduplication processing with at least one first position information to obtain at least one first detection position information; or, according to an inverse matrix corresponding to the coordinate conversion matrix, converting at least one first position information into the second coordinate space for matching and deduplication processing with at least one second position information to obtain at least one second detection position information, and obtaining at least one first detection position information based on the at least one second detection position information.

[0086] In the embodiment of the present disclosure, the coordinate conversion matrix can be used to convert the second coordinate point in the second coordinate space into the first coordinate space. It can be understood that if the first coordinate point in the first coordinate space needs to be converted into the second coordinate space, the inverse matrix of the coordinate conversion matrix can be used to convert the first coordinate point. In addition, for the method of obtaining the coordinate conversion matrix, refer to the above-mentioned relevant instructions and will not be repeated here. In actual processing, the first position information and the second position information can be aligned to the first coordinate space or the second coordinate space based on the coordinate conversion matrix or the inverse matrix corresponding to the coordinate conversion matrix for matching and deduplication processing, that is, the two types of position information are deduplicated. If the matching and deduplication processing is performed in the first coordinate space, the first detection position information can be directly obtained. If the matching and deduplication processing is performed in the second coordinate space, the second detection position information obtained can be converted again to obtain the first detection position information. The specific implementation method is not limited here.

[0087] In some embodiments, the first predicted position information in the above step S1 can be predicted based on a preset position predictor, which can be, for example, a Kalman filter. For example, in the case of continuous tracking and detection of the target object directly based on the first image, i.e., the thermal imaging image, the predicted position of the target object in the first image can be predicted based on the preset position predictor, thereby obtaining the first predicted position information; and in the case of continuous tracking and detection of the target object based on images of two types of modalities, namely, the first image and the second image, in order to facilitate user viewing, when the first predicted position information is predicted based on the preset position predictor, the second predicted position information of the target object in the second image can be first predicted, and then the first predicted position information can be obtained by converting the second predicted position information. The specific implementation form is not particularly limited here.

[0088] In addition, in this embodiment, after determining the target position information of the target object, the method may further include: updating the motion state parameters of a preset position predictor according to the target position information, wherein the updated motion state parameters are used by the preset position predictor to predict the position information of the target object at the next moment.

[0089] That is, in actual implementation, a predictor such as a Kalman filter can be used to predict the position, speed and other state data of the target object, for example, to predict the predicted position of the target object at the current moment. After determining the accurate target position of the target object at the current moment, the motion state data corresponding to the target object in the Kalman filter, such as position, speed and other data, can be updated based on the target position, so that the Kalman filter can continue to predict the position information of the target object at the next moment based on the updated motion state data, so as to avoid a large deviation between the predicted position and the actual position of the target object, which affects the continuous tracking effect of the target object.

[0090] According to the above description, in the embodiments of the present disclosure, taking into account the problem that in the related art, when directly detecting the position information of the target object based on the visible light image, the detection frame may be lost due to the lighting conditions or other objects blocking it, and thus the detection position of the target object cannot be determined; in the embodiments of the present disclosure, images of other modalities, namely the first image, are introduced to assist in obtaining the observed position of the target object, and by fusing at least one first position information and at least one second position information, the detection position information that can accurately represent the detection position of the target object is obtained. At the same time, in the case where the detection position of the target object is lost based on the image of any modality, the position information detected by the image of other modalities can be used to supplement it to avoid completely losing the target object, that is, the problem of losing the target when performing target tracking detection based on a single modality image is avoided by a two-way blind spot filling method, so as to reduce the difficulty of tracking trend prediction of the target object and improve the accuracy of the target position information determined.

[0091] Second, refer to Figure 6 , an embodiment of the present disclosure provides an electronic device, comprising:

[0092] One or more processors 101;

[0093] A memory 102, on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement the target determination method described in the first aspect of the embodiment of the present disclosure;

[0094] One or more I / O interfaces 103 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0095] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU) and the like; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus) and the like.

[0096] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0097] Thirdly, refer to Figure 7An embodiment of the present disclosure provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the target determination method described in the first aspect of the embodiment of the present disclosure is implemented.

[0098] Example

[0099] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present disclosure, the technical solutions provided by the embodiments of the present disclosure are described in detail below through specific embodiments:

[0100] The target determination method provided in the embodiment of the present disclosure can be applied to determine the position of traffic targets on the road, such as roads, pedestrians, etc., to achieve continuous tracking and detection of the target, see Figure 8 In this embodiment, the first image, ie, the thermal imaging image, can be acquired by a thermal imager, and the second image, ie, the visible light image, can be acquired by a camera.

[0101] First, the internal parameters of the thermal imager and the camera can be calibrated, and after the thermal imager and the camera are fixed, the external parameters of the two can be jointly calibrated. Figure 8 As shown, the two can be used to capture the same scene image to establish a correspondence between the coordinate space 1 corresponding to the first image captured by the thermal imager and the coordinate space 2 corresponding to the second image captured by the camera, that is, to obtain a coordinate transformation matrix between coordinate space 1 and coordinate space 2. The coordinate transformation matrix can be used, for example, to transform coordinate points in coordinate space 2 into coordinate space 1.

[0102] Secondly, after completing the calibration processing of the internal parameters and external parameters of the thermal imager and the camera, the thermal imager can be used to collect the second image corresponding to the target object, and the thermal imager can be used to collect the first image corresponding to the target object; wherein the target object can be, for example, Figure 8 At least one object such as a vehicle, a pedestrian, etc. shown in FIG.

[0103] Secondly, the first image and the second image can be input into the target detection model to perform target object detection processing to obtain at least one detection frame corresponding to the target object, so as to obtain at least one first detection position information corresponding to the target object according to the at least one detection frame; wherein, in this step, by performing target detection processing on the first image, at least one first detection frame, i.e., first position information, can be obtained, and by performing target detection processing on the second image, at least one second detection frame, i.e., second position information, can be obtained, and by performing target detection frame fusion on the at least one first detection frame and the at least one second detection frame, the at least one first detection frame corresponding to the target object can be obtained. Detection position information, wherein the detection frame fusion processing may be to convert the first detection frame and the second detection frame to the same coordinate space, for example, by converting the first position information corresponding to the first detection frame to the second position information corresponding to the second detection frame in the coordinate space 2 corresponding to the second image, and performing the fusion processing to obtain at least one second detection position information, and then converting the second detection position information to obtain the first detection position information, or, it may be to convert the second position information corresponding to the second detection frame to the first position information corresponding to the first detection frame in the coordinate space 1 corresponding to the first image, and perform the fusion processing to directly obtain the first detection position information. In addition, in this step, the prediction frame corresponding to the target object in the second image, that is, the second prediction position information, may also be predicted by the Kalman filter corresponding to the target object.

[0104] Secondly, the visible light image feature 1 and the thermal imaging feature 1 corresponding to the target object can be extracted according to the second predicted position information, and the visible light image feature 2 and the thermal imaging feature 2 corresponding to the target object can be extracted according to the at least one first detected position information;

[0105] Among them, when extracting the thermal imaging feature 1 corresponding to the target object according to the second predicted position information, the second predicted position information can be converted into the coordinate space 1 corresponding to the first image to obtain the first predicted position information, and the thermal imaging feature 1 can be extracted from the position represented by the first predicted position information in the first image; when extracting the thermal imaging feature 2 corresponding to the target object according to the at least one first detection position information, the thermal imaging feature 2 can be extracted from the position represented by the at least one first detection position information in the first image. Of course, if only the second detection position information is obtained when the above-mentioned first position information and second position information are matched and deduplicated, the second detection position information can also be converted here to obtain the first detection position information, and then the thermal imaging feature 2 can be extracted according to the first detection position information. The specific implementation form is not limited here.

[0106] Afterwards, the visible light image feature 1 can be matched with at least one visible light image feature 2 to obtain a matching result 1, and the thermal imaging feature 1 can be matched with at least one visible light image feature 2 to obtain a matching result 2. Based on the matching result 1 and the matching result 2 combined with the Hungarian algorithm, the target position information of each object in the currently collected visible light image is determined to achieve continuous tracking and detection of the target object.

[0107] Finally, if Figure 8 As shown, after determining the position information of each target object, each target object may be assigned a target ID, and the motion state data in the Kalman filter may be updated.

[0108] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0109] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A target determination method, characterized in that: include: Acquire first predicted position information and at least one first detected position information of a target object; Obtaining first feature information corresponding to the target object according to the first predicted position information, and obtaining at least one second feature information corresponding to the target object according to the at least one first detected position information; wherein the first feature information and the second feature information include thermal imaging features of the target object; The target position information of the target object is determined according to the first feature information and the at least one second feature information.

2. The method according to claim 1, characterized in that The obtaining, according to the first predicted position information, first feature information corresponding to the target object, and the obtaining, according to the at least one first detected position information, at least one second feature information corresponding to the target object, include: extracting a first thermal imaging feature from a position corresponding to the first predicted position information in the first image, and obtaining the first feature information according to the first thermal imaging feature, wherein the first image is a thermal imaging image including the target object; At least one second thermal imaging feature is extracted from a position corresponding to the at least one first detection position information in the first image, and the at least one second feature information is obtained according to the at least one second thermal imaging feature.

3. The method according to claim 2, characterized in that The determining the target location information of the target object according to the first feature information and the at least one second feature information includes: Matching the first thermal imaging feature with the at least one second thermal imaging feature to obtain a first matching result; The target location information is determined according to the first matching result.

4. The method according to claim 2, characterized in that: The first predicted position information is obtained according to the second predicted position information, the second predicted position information is used to indicate the predicted position of the target object in the second image, the at least one first detected position information includes position information obtained according to the at least one second detected position information, the second detected position information is used to indicate the detected position of the target object in the second image, and the second image includes other images except the thermal imaging image; The extracting of a first thermal imaging feature from a position corresponding to the first predicted position information in the first image, and extracting of at least one second thermal imaging feature from a position corresponding to the at least one first detected position information in the first image, comprises: Obtaining a coordinate transformation matrix between a first coordinate space corresponding to the first image and a second coordinate space corresponding to the second image; According to the coordinate conversion matrix, performing position conversion processing on the second predicted position information and the at least one second detected position information respectively, to obtain the first predicted position information corresponding to the second predicted position information, and the at least one first detected position information corresponding to the at least one second detected position information; The first thermal imaging feature is extracted from a position in the first image represented by the first predicted position information, and the at least one second thermal imaging feature is extracted from a position in the first image represented by the at least one first detected position information.

5. The method according to claim 4, characterized in that The second image is a visible light image; the first feature information further includes a first visible light image feature, which is a feature extracted from the position represented by the second predicted position information in the second image; the second feature information further includes a second visible light image feature, which is a feature extracted from the position represented by the second detected position information in the second image; The determining the target location information of the target object according to the first feature information and the at least one second feature information includes: Matching the first thermal imaging feature with the at least one second thermal imaging feature to obtain a first matching result; Matching the first visible light image feature with the at least one second visible light image feature to obtain a second matching result; The target location information of the target object is determined according to the first matching result and the second matching result.

6. The method according to claim 1, characterized in that The obtaining of at least one first detected position information of the target object comprises: Acquire a first image and a second image corresponding to the target object, wherein the first image includes a thermal imaging image, and the second image includes an image other than the thermal imaging image; Acquire at least one first position information corresponding to the target object in the first image, and acquire at least one second position information corresponding to the target object in the second image; The at least one first detection position information is obtained according to the at least one first position information and the at least one second position information.

7. The method according to claim 6, characterized in that The obtaining, according to the at least one first position information and the at least one second position information, the at least one first detection position information comprises: The at least one first position information and the at least one second position information are matched to obtain the at least one first detection position information.

8. The method according to claim 7, characterized in that The matching process of the at least one first position information and the at least one second position information to obtain the at least one first detection position information includes: The at least one first position information and the at least one second position information are converted into the same coordinate space for matching and deduplication processing to obtain the at least one first detection position information.

9. The method according to claim 8, characterized in that The first position information is in a first coordinate space corresponding to the first image, and the second position information is in a second coordinate space corresponding to the second image; The converting the at least one first position information and the at least one second position information into the same coordinate space for matching and deduplication processing to obtain the at least one first detection position information includes: Obtaining a coordinate transformation matrix corresponding to the first coordinate space and the second coordinate space; According to the coordinate conversion matrix, the at least one second position information is converted into the first coordinate space to perform matching and deduplication processing with the at least one first position information to obtain the at least one first detection position information; or, According to the inverse matrix corresponding to the coordinate transformation matrix, the at least one first position information is converted to the second coordinate space and matched with the at least one second position information for deduplication processing to obtain at least one second detection position information, and the at least one first detection position information is obtained based on the at least one second detection position information.

10. The method according to claim 6, characterized in that The acquiring at least one first position information of the target object in the first image, and acquiring at least one second position information of the target object in the second image, comprises: Performing detection processing on the first image to obtain at least one first detection frame corresponding to the target object, and obtaining the at least one first position information according to the at least one first detection frame; The detection process is performed on the second image to obtain at least one second detection frame corresponding to the target object, and the at least one second position information is obtained based on the at least one second detection frame.

11. The method according to any one of claims 1 to 10, characterized in that The first predicted position information is predicted by a preset position predictor, and the preset position predictor includes a Kalman filter; The method further comprises: According to the target position information, the motion state parameters of the preset position predictor are updated, wherein the updated motion state parameters are used by the preset position predictor to predict the position information of the target object at the next moment.

12. An electronic device comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the target determination method according to any one of claims 1 to 11.

13. A computer readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the target determination method according to any one of claims 1 to 11 is implemented.

Images