An image stabilization method, device, electronic equipment and storage medium

Abstract

The present disclosure provides an image stabilization method and device, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to the technical field of computer vision, intelligent transportation, vehicle-road cooperation and the like. The specific implementation scheme is: detecting the current image foreground of a perception device to obtain information of each obstacle in the current image; for each obstacle, determining a corresponding target stabilization frame in each preset stabilization frame of the current image; determining whether the obstacle shields the corresponding target stabilization frame based on the information of the obstacle; when not shielding, calculating the offset between the corresponding target stabilization frame and the corresponding stabilization frame in the reference image to obtain the stabilization offset of the corresponding target stabilization frame of the obstacle; when the stabilization offset of the corresponding target stabilization frame of at least one obstacle does not exceed a preset threshold, determining the new position of each obstacle based on the stabilization offset of the corresponding target stabilization frame of at least one obstacle and the information of each obstacle, thereby realizing the stabilization of the perception device.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of computer vision, intelligent transportation, and vehicle-road cooperation, and specifically to an image stabilization method, apparatus, electronic device, and storage medium. Background Technology

[0002] Roadside perception is the core of vehicle-road cooperation. It captures and analyzes data by installing cameras and other sensing devices on the roadside. If the sensing devices undergo slight or large displacements while operating online, the relative position of the object captured by the sensing devices will change from the initial calibration position of the object. Therefore, it is necessary to stabilize the image of the object captured by the sensing devices to correct the positional shift of the object captured by the sensing devices. Summary of the Invention

[0003] This disclosure provides an image stabilization method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, an image stabilization method is provided, comprising:

[0005] Foreground detection is performed on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image;

[0006] For each obstacle, a target image stabilization frame corresponding to the obstacle is determined in each preset image stabilization frame of the current image;

[0007] Based on the information about the obstacle, determine whether the obstacle occludes the target frame corresponding to the obstacle;

[0008] If the obstacle does not obscure the target frame corresponding to the obstacle, calculate the offset between the target frame corresponding to the obstacle and the corresponding frame in the reference image to obtain the image stabilization offset of the target frame corresponding to the obstacle.

[0009] If the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the image stabilization offset of the target image stabilization frame corresponding to the at least one obstacle and the information of each obstacle.

[0010] According to another aspect of this disclosure, an image stabilization apparatus is provided, comprising:

[0011] The foreground detection module is used to perform foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image;

[0012] The image stabilization frame determination module is used to determine the target image stabilization frame corresponding to each obstacle in each preset image stabilization frame of the current image;

[0013] The foreground occlusion determination module is used to determine whether the obstacle occludes the target frame corresponding to the obstacle based on the information of the obstacle;

[0014] The image stabilization offset calculation module is used to calculate the offset between the target image stabilization frame corresponding to the obstacle and the corresponding image stabilization frame in the reference image when the obstacle does not obscure the target image stabilization frame corresponding to the obstacle, and obtain the image stabilization offset of the target image stabilization frame corresponding to the obstacle.

[0015] The position determination module is used to determine the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle and the information of each obstacle, provided that the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform any of the image stabilization methods described in this disclosure.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are configured to cause the computer to perform the image stabilization method described in any one of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the image stabilization method described in any one of this disclosure.

[0022] In this embodiment of the disclosure, image stabilization of the sensing device is achieved.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0024] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0025] Figure 1 This is a schematic diagram of an image stabilization method according to the present disclosure;

[0026] Figure 2 This is a schematic diagram of the method for determining the image stabilization offset according to this disclosure;

[0027] Figure 3 This is a schematic diagram of an overall framework for the image stabilization method according to the present disclosure;

[0028] Figure 4 Another schematic diagram of the image stabilization method according to this disclosure;

[0029] Figure 5 This is a schematic diagram of another overall framework for the image stabilization method according to this disclosure;

[0030] Figure 6 This is a schematic diagram of an image stabilization device according to the present disclosure;

[0031] Figure 7 This is a block diagram of an electronic device used to implement the image stabilization method of the embodiments of this disclosure. Detailed Implementation

[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] In a roadside object recognition scenario, after the roadside sensing device is installed, the position of the object to be recognized remains relatively unchanged relative to the camera of the roadside sensing device. Therefore, an image containing the object to be recognized can be pre-captured by the camera of the sensing device, and the position of the object to be recognized can be manually labeled to complete the parameter calibration of the sensing device. The object to be recognized can be, for example, a traffic light or a roadside building, and the sensing device can be, for example, a roadside camera.

[0034] However, in real-world scenarios, after roadside sensing devices are installed and calibrated, their pose may change due to various factors, such as stress relief during installation, gravity-induced tilting, large vehicles passing by, or vibrations caused by strong winds. Once the sensor's pose changes, the previously calibrated parameters can no longer accurately describe it, leading to a decrease in sensing accuracy.

[0035] In related technologies, manual on-site recalibration and reinstallation of the sensing equipment are used to solve the problems caused by changes in the pose of the sensing equipment.

[0036] To achieve image stabilization in a sensing device, the image stabilization method provided in this disclosure performs foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image. For each obstacle, a target stabilization frame corresponding to the obstacle is determined in each preset stabilization frame of the current image. Based on the information of the obstacle, it is determined whether the obstacle occludes the target stabilization frame corresponding to the obstacle. If the obstacle does not occlude the target stabilization frame corresponding to the obstacle, the offset between the target stabilization frame corresponding to the obstacle and the corresponding stabilization frame in the reference image is calculated to obtain the stabilization offset of the target stabilization frame corresponding to the obstacle. If the stabilization offset of the target stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the stabilization offset of the target stabilization frame corresponding to at least one obstacle and the information of each obstacle.

[0037] In this embodiment, foreground detection is performed on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image. Each obstacle is then traversed, and a target stable frame corresponding to each obstacle is determined within each preset stable frame of the current image. It is then determined whether each obstacle occludes its corresponding target stable frame. If not occluded, the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image is calculated to obtain the stable offset of the target stable frame corresponding to each obstacle. If the stable offset of the target stable frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the stable offset of the target stable frame corresponding to at least one obstacle and the information of each obstacle. This achieves real-time online image stabilization of the sensing device in scenarios where the sensing device undergoes small displacement, reducing the cost of manual on-site recalibration and reinstallation of the sensing device.

[0038] The image stabilization method provided in the embodiments of this disclosure will be described in detail below.

[0039] The image stabilization method provided in this disclosure can be applied to electronic devices, such as server devices, smart terminal devices, or roadside sensing devices, etc.

[0040] See Figure 1 , Figure 1 A flowchart illustrating an image stabilization method provided in this disclosure includes the following steps:

[0041] S101, perform foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image.

[0042] In one example, during the online operation of the sensing device, a target detection model is used to perform foreground detection on the current image acquired by the sensing device to obtain information about various obstacles contained in the current image. These obstacles can be, for example, pedestrians, motor vehicles, non-motorized vehicles, and traffic lights, while the sensing device can be, for example, a roadside camera, a roadside video camera, or a roadside network camera.

[0043] The object detection model can be trained based on sample images and information about obstacles within those images. The obstacle information can include the obstacle's identifier, the location of the corresponding bounding box in the current image, its center coordinates, and so on.

[0044] S102, for each obstacle, determine the target image frame corresponding to the obstacle in each preset image frame of the current image.

[0045] In one example, after installing the sensing device, the first image acquired by the device can be used as a reference image. All pixels in this reference image are used as reference pixels. Specific objects within the reference image are then calibrated (e.g., using an object detection model to detect specific objects and further calibrating the detection boxes of those objects). The calibrated detection boxes serve as reference stabilized frames. These specific objects could be, for example, traffic lights, roadside buildings, etc. Each reference stabilized frame in the reference image can be mapped to a preset stabilized frame in each image captured by the sensing device.

[0046] After acquiring information about each obstacle in the current image, using each obstacle as a reference, the system iterates through each obstacle and determines the target stable frame corresponding to that obstacle within each preset stable frame in the current image. In one example, the distance between each obstacle and each preset stable frame in the current image can be calculated, and the preset stable frame closest to the obstacle can be determined as the target stable frame for that obstacle. Alternatively, preset stable frames whose distance is within a preset distance threshold can be determined as the target stable frame for that obstacle. The preset distance threshold can be set according to actual needs.

[0047] S103, based on the information of the obstacle, determine whether the obstacle occludes the target image frame corresponding to the obstacle.

[0048] After determining the corresponding target stable bounding box for each obstacle, it is determined whether the obstacle occludes its corresponding target stable bounding box. In one example, the obstacle information is the obstacle's position information, which allows the determination of the coordinates of the detection box corresponding to the obstacle in the current image. These coordinates are then matched with the coordinates of the target stable bounding box to detect overlap. If overlap is found, the obstacle occludes its target stable bounding box; otherwise, it does not. Alternatively, the pixels of the obstacle in the current image are matched with the pixels of the target stable bounding box to detect overlap. If overlap is found, the obstacle occludes its target stable bounding box; otherwise, it does not.

[0049] S104, if the obstacle does not obscure the target image frame corresponding to the obstacle, calculate the offset between the target image frame corresponding to the obstacle and the corresponding image frame in the reference image, and obtain the image stabilization offset of the target image frame corresponding to the obstacle.

[0050] If the obstacle does not obscure the target frame corresponding to the obstacle, the offset between the pixels of the target frame corresponding to the obstacle and the pixels of the corresponding frame in the reference image can be calculated to obtain the image stabilization offset of the target frame corresponding to the obstacle.

[0051] S105, if the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold, determine the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle and the information of each obstacle.

[0052] After obtaining the image stabilization offset of the target image stabilization frame corresponding to each obstacle, the relationship between the image stabilization offset of the target image stabilization frame corresponding to each obstacle and the preset offset threshold is further determined. If the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed the preset offset threshold, the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle is used to correct the information of each obstacle and determine the new position of each obstacle in order to stabilize the current image.

[0053] The preset offset threshold can be set according to actual conditions. In one example, the preset offset threshold can be used to measure whether the displacement of the sensing device is recoverable, or whether the image acquired by the sensing device can be recovered without changing the reference image. If the image stabilization offset of the target frame corresponding to at least one obstacle does not exceed the preset offset threshold, it indicates that the sensing device has undergone a small displacement, and the image acquired by the sensing device can be stabilized without changing the reference image. If the image stabilization offset of the target frames corresponding to all obstacles in the current image exceeds the preset offset threshold, it indicates that the sensing device has undergone a large displacement, and it may be necessary to recalibrate the extrinsic parameters of the sensing device.

[0054] In this embodiment, foreground detection is performed on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image. Each obstacle is then traversed, and a target stable frame corresponding to each obstacle is determined within each preset stable frame of the current image. It is then determined whether each obstacle occludes its corresponding target stable frame. If not occluded, the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image is calculated to obtain the stable offset of the target stable frame corresponding to each obstacle. If the stable offset of the target stable frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the stable offset of the target stable frame corresponding to at least one obstacle and the information of each obstacle. This achieves real-time online image stabilization of the sensing device in scenarios where the sensing device undergoes small displacement, reducing the cost of manual on-site recalibration and reinstallation of the sensing device.

[0055] In one possible implementation, step S104 above involves calculating the offset between the target image-stabilized frame corresponding to the obstacle and the corresponding image-stabilized frame in the reference image, thus obtaining the image-stabilized offset of the target image-stabilized frame corresponding to the obstacle. Figure 2 As shown, it may include:

[0056] S201, calculate the offset between the target frame corresponding to the obstacle and the corresponding frame in the reference image to obtain the target offset.

[0057] In one example, the offset between the pixels of the target frame corresponding to the obstacle and the pixels of the corresponding frame in the reference image can be calculated, and the minimum or average value of the calculated offset is determined as the target offset.

[0058] S202, calculate the similarity between the current image and the reference image.

[0059] In one example, the SSIM (Structural Similarity Index measure) between the current image and the reference image can be calculated, or the PSNR (Peak Signal-to-Noise Ratio) between the current image and the reference image can be calculated, and then the similarity between the current image and the reference image can be measured by the SSIM or PSNR between the current image and the reference image.

[0060] The smaller the PSNR between the current image and the reference image, the more similar the current image and the reference image are. Typically, the SSIM value ranges from [0,1]. The closer the SSIM between the current image and the reference image is to 1, the more similar the current image and the reference image are.

[0061] S203, if the similarity meets the preset conditions, the target offset is determined as the image stabilization offset of the target image frame corresponding to the obstacle, and the target offset is added to the historical offset list.

[0062] When measuring the similarity between the current image and the reference image using SSIM or PSNR, preset conditions can be set accordingly. For example, if the calculation is of SSIM between the current image and the reference image, the preset condition can be set to no less than 0.7, 0.8, or 0.9. If the calculation is of PSNR between the current image and the reference image, the preset condition can be set to no greater than 0.1, 0.2, or 0.3.

[0063] If the similarity between the current image and the reference image meets the preset conditions, it indicates that the current image is similar to the reference image. At this time, the calculated target offset is considered to be relatively accurate. The target offset is determined as the image stabilization offset of the target image frame corresponding to the obstacle, and the target offset is added to the historical offset list.

[0064] S204, if the similarity does not meet the preset conditions, obtain each historical offset from the historical offset list, and determine the average value of each historical offset as the image stabilization offset of the target image frame corresponding to the obstacle.

[0065] If the similarity between the current image and the reference image does not meet the preset conditions, it indicates that the current image and the reference image are not similar. In this case, the calculated target offset is considered relatively inaccurate. Therefore, each historical offset is obtained from the historical offset list, and the average of each historical offset is determined as the image stabilization offset of the target frame corresponding to the obstacle. Furthermore, the average of each historical offset can be added to the historical offset list.

[0066] In one example, at least two historical offsets can be preset in the historical offset list to ensure that even if the current image is the first image of the stabilized image and the similarity between the current image and the reference image does not meet the preset conditions, each historical offset can still be obtained from the historical offset list, thereby determining the image stabilization offset of the target image stabilization box corresponding to the obstacle.

[0067] In one example, the length of the historical offset list can also be set. After a set number of target offsets are added to the historical offset list, the earliest target offset added to the historical offset list is deleted each time a new target offset is added, so as to avoid interference from the target offsets added earlier and improve the calculation accuracy of the image stabilization offset of the target image frame corresponding to the obstacle.

[0068] In this embodiment of the present disclosure, when the current image is similar to the reference image, the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image is determined as the stable offset of the target stable frame corresponding to the obstacle. When the current image is not similar to the reference image, the average value of each historical offset in the historical offset list is determined as the stable offset of the target stable frame corresponding to the obstacle, so as to accurately calculate the stable offset of the target stable frame corresponding to the obstacle.

[0069] In one possible implementation, the image stabilization method described above may further include:

[0070] When an obstacle obscures the target frame corresponding to the obstacle, each historical offset is obtained from the historical offset list, and the average of each historical offset is determined as the image stabilization offset of the target frame corresponding to the obstacle.

[0071] If, in step S103 above, it is determined that the obstacle occludes the target stable frame corresponding to the obstacle, then the pixel points of the target stable frame corresponding to the obstacle cannot be accurately obtained, and therefore the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image cannot be accurately calculated. In this case, each historical offset is obtained from the historical offset list, and the average of each historical offset is determined as the stable offset of the target stable frame corresponding to the obstacle. Furthermore, the average of each historical offset can also be added to the historical offset list.

[0072] In this embodiment of the present disclosure, even when an obstacle obscures the target frame corresponding to the obstacle, the image stabilization offset of the target frame corresponding to the obstacle can still be determined based on each historical offset in the historical offset list, thereby achieving image stabilization.

[0073] In one possible implementation, the information about the obstacles is their location information. Accordingly, step S105, which determines the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle and the information of each obstacle, may include:

[0074] Calculate the homography matrix or interpolation matrix from the stable frame of the reference image to the corresponding stable frame of the current image; based on the position information of each obstacle, and the homography matrix or interpolation matrix, determine the new position of each obstacle.

[0075] In one example, taking any stable frame as an example, based on the stable frame position of the reference image and the corresponding stable frame position of the current image, the homography matrix or interpolation matrix from the stable frame of the reference image to the corresponding stable frame of the current image is calculated. Further, for each obstacle, the position of the obstacle is calculated, and the product of the obstacle's position and the inverse of the aforementioned homography matrix or interpolation matrix is ​​used to obtain the new position of each obstacle in the current image.

[0076] The stable frame position of the current image can be obtained by detecting the target using a target detection model, or it can be calculated based on the stabilized frame offset of each obstacle as calculated above, and the stabilized frame position of the reference image. In one example, the average of the historical offsets in the historical offset list can be used as the offset between the stabilized frame of the current image and the reference image. Furthermore, the stabilized frame position of the current image is obtained by subtracting the offset between the stabilized frame of the current image and the reference image from the stabilized frame position of the reference image.

[0077] In this embodiment of the present disclosure, the homography matrix or interpolation matrix from the stabilized frame of the reference image to the corresponding stabilized frame of the current image is calculated. Then, based on the position information of each obstacle and the calculated homography matrix or interpolation matrix, the position of each obstacle in the current image is accurately corrected to achieve image stabilization.

[0078] In one possible implementation, when the obstacle is a traffic light, if the calculated image stabilization offset of the target image frame corresponding to the traffic light is greater than a set value, it is determined that the traffic light may be displaced or shaking, and an alarm message can be generated to alert the traffic light to the displacement. This set value can be set according to actual needs.

[0079] In one possible implementation, the above also includes:

[0080] If the image stabilization offset of the target stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold, a displacement alarm will be triggered by the sensing device.

[0081] In one example, if the image stabilization offset of the target stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold, or if the minimum value of the image stabilization offset of the target stabilization frame corresponding to each obstacle exceeds the preset offset threshold, it indicates that the sensing device may have undergone a large displacement. In this case, without changing the reference image, it is impossible to accurately stabilize the image acquired by the sensing device. It may be necessary to recalibrate the external parameters of the sensing device and generate alarm information to alert the sensing device to the displacement.

[0082] In this embodiment of the present disclosure, when the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds a preset offset threshold, a displacement alarm of the sensing device is triggered to promptly indicate that the sensing device may have undergone a large displacement.

[0083] For example, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the overall framework for image stabilization. Roadside sensing devices, terminals, or servers, among other electronic devices, perform foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image. In one example, the obstacle can be any Roi (region of interest).

[0084] For each obstacle, within each preset image stabilization frame of the current image, a target image stabilization frame corresponding to the obstacle is determined. Based on the detected obstacle information, it is determined whether the obstacle occludes the target image stabilization frame corresponding to the obstacle (i.e., ...). Figure 3 (Mid-foreground occlusion judgment).

[0085] If the obstacle does not obscure the target frame corresponding to the obstacle, calculate the offset between the target frame corresponding to the obstacle and the corresponding frame in the reference image to obtain the target offset (specifically, this can be achieved using...). Figure 3 The target offset is calculated using a mid-pixel / subpixel registration method, and the similarity between the current image and the reference image is calculated (i.e., Figure 3 In the image similarity assessment, if the similarity meets a preset condition (i.e., the similarity assessment passes), the target offset is determined as the stabilized image offset of the target frame corresponding to the obstacle, and the target offset is added to the historical offset queue. If the similarity does not meet the preset condition (i.e., the similarity assessment does not pass), each historical offset is obtained from the historical offset queue, and the average of each historical offset is determined as the stabilized image offset of the target frame corresponding to the obstacle, and the average of each historical offset is added to the historical offset queue.

[0086] When an obstacle occludes the target frame corresponding to that obstacle, all historical offsets are directly retrieved from the historical offset queue. The average of these historical offsets is then used as the stabilized offset of the target frame corresponding to that obstacle, and the average of all historical offsets is added to the historical offset queue. Specifically, the historical offset queue contains the stabilized offsets of the target frames corresponding to each obstacle. Figure 3 In this context, bias is represented as bias1, bias2, ..., biasN.

[0087] Based on the above method, the image stabilization offset of the target image stabilization frame corresponding to each obstacle is obtained. Figure 3 The values ​​are represented as Roi1 offset, Roi2 offset, Roi3 offset, and Roi4 offset (in practice, there may be more or fewer than four). Further, if the image stabilization offset of the target image frame corresponding to at least one obstacle does not exceed a preset offset threshold, the homography matrix or interpolation matrix from the stabilized image frame of the reference image to the corresponding stabilized image frame of the current image is calculated. Based on the position information of each obstacle and the homography matrix or interpolation matrix, the new position of each obstacle is determined. When the obstacle is a traffic light, if the calculated image stabilization offset of the target image frame corresponding to the traffic light is greater than a set value, it is determined that the traffic light may have shifted or shaken, and an alarm is triggered for the traffic light shift. Additionally, if the image stabilization offset of the target image frames corresponding to all obstacles in the current image exceeds the preset offset threshold, a displacement alarm for the sensing device is triggered.

[0088] In practical applications, most of the displacements of online sensing devices are small to medium amplitude vibrations. The image stabilization method described above can achieve real-time online image stabilization of the sensing devices. However, there are also a small number of sensing devices that experience large displacements, in which case it may be necessary to recalibrate the external parameters of the sensing devices.

[0089] In one possible implementation, based on the above-described image stabilization method, it may further include:

[0090] If the image stabilization offset of the target stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold, the current image is added to the image list; if the number of images in the image list meets the preset extrinsic calibration conditions, the extrinsic calibration value of the sensing device is re-determined.

[0091] Based on the image stabilization offset of the target frame corresponding to each obstacle in the current image, further judgment is made. If the image stabilization offset of the target frame corresponding to each obstacle in the current image exceeds a preset offset threshold, a displacement alarm is triggered on the sensing device, and the current image is added to the image list. If the number of images in the image list meets a preset extrinsic parameter calibration condition, the extrinsic parameter calibration value of the sensing device is redefined. The preset extrinsic parameter calibration condition can be that the number of images in the image list reaches a preset value, which can be set according to actual needs.

[0092] In this embodiment of the disclosure, when the number of images in which the image stabilization offset of the target image stabilization frame corresponding to each obstacle exceeds the preset offset threshold, and the preset extrinsic parameter calibration conditions are met, the extrinsic parameter calibration value of the sensing device is re-determined to more accurately recalibrate the extrinsic parameters of the sensing device. This avoids the need to recalibrate the extrinsic parameters of the sensing device when it recovers after a large displacement caused by external factors.

[0093] In one possible implementation, such as Figure 4 As shown, the method for redetermining the external parameter calibration values ​​of the sensing device may include the following steps:

[0094] S401, retrieve a preset number of keyframe images from the image list.

[0095] The preset number can be set according to actual needs. The keyframe image is the image in which the image stabilization offset of the target image stabilization frame corresponding to each obstacle obtained above all exceeds the preset offset threshold.

[0096] S402, For each keyframe image, calculate the offset of the stabilized frame of the keyframe image from the corresponding stabilized frame of the reference image, and obtain the offset of the stabilized frame of the keyframe image.

[0097] For each keyframe image, taking any stable frame as an example, the offset between the stable frame of each keyframe image and the corresponding stable frame of the reference image is calculated based on the pixels of the stable frame of the keyframe image and the pixels of the stable frame of the reference image, thus obtaining the offset of the stable frame of each keyframe image.

[0098] S403, based on the offset of the stabilization frame of each keyframe image, determines whether it is necessary to re-determine the external parameter calibration values ​​of the sensing device.

[0099] In one example, after obtaining the offset of the stable frame of each keyframe image, the number of keyframe images with an offset greater than a preset offset threshold is determined. If this number exceeds half, two-thirds, or three-quarters of the total number of keyframe images, it is considered that the external parameter calibration value of the sensing device needs to be re-determined.

[0100] S404, when it is determined that the external parameter calibration values ​​of the sensing device need to be redefined, select the target keyframe image from the keyframe images.

[0101] In one example, a keyframe image can be randomly selected as the target keyframe image, or the keyframe image with the smallest offset of the stable frame can be selected as the target keyframe image, etc.

[0102] S405, based on the target keyframe image and a pre-determined reference mapping table, redetermine the extrinsic calibration values ​​of the sensing device.

[0103] The reference mapping table contains images from different sensing devices at different viewpoints, positions, and orientations, as well as the pixel coordinates of those images.

[0104] The pixel coordinates of the selected target keyframe image are matched with the pixel coordinates of each image in a pre-determined reference mapping table to obtain the target image whose pixel coordinates match those of the target keyframe image in the reference mapping table. The extrinsic parameters of the sensing device corresponding to this target image are then determined as the new extrinsic parameter calibration values ​​for the sensing device. These extrinsic parameters include the sensing device's viewpoint, position, and orientation.

[0105] The above Figure 4 It can be executed alone, or in... Figure 1 Executed on the basis of [previous]. Figure 4 When executed independently, for each keyframe image in the image list (a preset number of keyframe images), the offset of the stabilized frame of that keyframe image from the corresponding stabilized frame of the reference image is calculated. Based on the offsets of the stabilized frames of each keyframe image, it is determined whether the extrinsic parameter calibration values ​​of the sensing device need to be redefined. If it is determined that the extrinsic parameter calibration values ​​of the sensing device need to be redefined, a target keyframe image is selected from the keyframe images. Based on the target keyframe image and a pre-determined reference mapping table, the extrinsic parameter calibration values ​​of the sensing device are redefined to enable redefined extrinsic parameter calibration values ​​in scenarios where the sensing device undergoes significant displacement. Figure 1 When the image stabilization method shown fails to achieve image stabilization, the external parameters of the sensing device should be accurately recalibrated.

[0106] Figure 4 exist Figure 1 When executed based on this, it can achieve real-time online image stabilization of the sensing device in scenarios with small displacements, reducing the costs associated with manual on-site recalibration and reinstallation of the sensing device. It can also achieve image stabilization in scenarios with large displacements of the sensing device. Figure 1 When the image stabilization method shown fails to achieve image stabilization, the external parameters of the sensing device should be accurately recalibrated.

[0107] In one possible implementation, step S402 above, calculating the offset between the stabilized frame of the keyframe image and the corresponding stabilized frame of the reference image to obtain the offset of the stabilized frame of the keyframe image, may include:

[0108] Using the pyramid feature tracking algorithm, the offset of the stable frame of the keyframe image from the corresponding stable frame of the reference image is calculated, and the first offset of the stable frame of the keyframe image is obtained.

[0109] Using the feature point matching method, the offset of the stable frame of the keyframe image from the corresponding stable frame of the reference image is calculated, and the second offset of the stable frame of the keyframe image is obtained.

[0110] Using the lane line matching method, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the third offset of the stable frame of the keyframe image is obtained.

[0111] In this embodiment of the disclosure, when computational resources permit, the offset of the stable frame of each keyframe image from the corresponding stable frame of the reference image is calculated using the pyramid feature tracking algorithm, the feature point matching method, and the lane line matching method, respectively. The offsets of the stable frames of each keyframe image are then cross-checked using these methods to ensure the accuracy of the extrinsic parameter calibration values ​​of the sensing device that need to be re-determined.

[0112] In the pyramid feature tracking algorithm, a pyramid is built between the keyframe image and the reference image. The pyramid is then calculated layer by layer using detection boxes of different preset scales to obtain the precise offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image.

[0113] In feature point matching or feature point / line matching methods, the SIFT (Scale Invariant Feature Transform) operator or the LSD (Line Segment Detector) line feature extraction operator is used to perform feature matching between the stable frame of the keyframe image and the corresponding stable frame of the reference image, so as to obtain the accurate offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image.

[0114] In the lane line matching method, 2D (two-dimensional) - 3D (three-dimensional) lane line corner feature matching is performed between the stable frame of the keyframe image and the corresponding stable frame of the reference image to obtain the accurate offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image.

[0115] Accordingly, in step S403 above, determining whether it is necessary to recalibrate the extrinsic parameters of the sensing device based on the offset of the stabilization frame of each keyframe image may include:

[0116] Based on the first offset of the stable frame of each key frame image calculated using the pyramid feature tracking algorithm, the second offset of the stable frame of each key frame image calculated using the feature point matching method, and the third offset of the stable frame of each key frame image calculated using the lane line matching method, it is determined whether the external parameter calibration values ​​of the sensing device need to be redefined.

[0117] If the first offset of the stabilized frame of each keyframe image calculated using the pyramid feature tracking algorithm, the second offset of the stabilized frame of each keyframe image calculated using the feature point matching method, and the third offset of the stabilized frame of each keyframe image calculated using the lane line matching method satisfy the set conditions, then it is determined that the extrinsic parameter calibration values ​​of the sensing device need to be re-determined; otherwise, it is determined that the extrinsic parameter calibration values ​​of the sensing device do not need to be re-determined. The set conditions can be, for example, setting a threshold.

[0118] In one example, the mean or minimum value of the first offset of the stabilized frame of each keyframe image is calculated to obtain the first deviation; the mean or minimum value of the second offset of the stabilized frame of each keyframe image is calculated to obtain the second deviation; and the mean or minimum value of the third offset of the stabilized frame of each keyframe image is calculated to obtain the third deviation.

[0119] If at least two of the first, second, and third deviations are greater than the aforementioned preset offset threshold, it is determined that the extrinsic parameter calibration values ​​of the sensing device need to be redefined. Alternatively, weights are assigned to the first offset of the stabilized frame of each keyframe image calculated using the pyramid feature tracking algorithm, the second offset of the stabilized frame of each keyframe image calculated using the feature point matching method, and the third offset of the stabilized frame of each keyframe image calculated using the lane line matching method. Then, the weighted sum of the first offset of the stabilized frame of each keyframe image is calculated to obtain a first weighted value. The weighted sum of the second offset of the stabilized frame of each keyframe image is calculated to obtain a second weighted value. The weighted sum of the third offset of the stabilized frame of each keyframe image is calculated to obtain a third weighted value. If the offset corresponding to the maximum value among the first, second, and third weighted values ​​is greater than the aforementioned preset offset threshold, it is determined that the extrinsic parameter calibration values ​​of the sensing device need to be redefined.

[0120] In this embodiment of the disclosure, the pyramid feature tracking algorithm, the feature point matching method, and the lane line matching method are used respectively to calculate the offset of the stable frame of each key frame image from the corresponding stable frame of the reference image, thereby obtaining the offset of the stable frame of each key frame image. Then, the offsets of the stable frames of each key frame image calculated by the pyramid feature tracking algorithm, the feature point matching method, and the lane line matching method are cross-checked to ensure the accuracy of the external parameter calibration values ​​of the sensing device that need to be re-determined.

[0121] In one possible implementation, the reference mapping table can be obtained through the following steps:

[0122] The target object is captured by multiple sensing devices to obtain multiple images of the target object from different sensing device perspectives, positions, and orientations. Each sensing device perspective, position, and orientation corresponds to one of the sensing devices in the multiple sensing devices.

[0123] A reference mapping table is generated based on multiple images of the target object taken from different perspectives, positions, and orientations of different sensing devices, as well as the pixel coordinates of the images.

[0124] The target objects could be traffic lights or roadside buildings, for example.

[0125] In one example, multiple sensing devices can be used in advance, each with a different viewpoint, position, and orientation. Each sensing device is then used to capture images of the target object, resulting in multiple images of the target object from different viewpoints, positions, and orientations. The viewpoint, position, and orientation parameters of the sensing device corresponding to each image are recorded. Furthermore, the identification information of the multiple target object images from different viewpoints, positions, and orientations, the viewpoint, position, and orientation parameters of the sensing devices, and the pixel coordinates of the images are correlated to generate a reference mapping table.

[0126] In this embodiment of the disclosure, the target object is captured by multiple sensing devices to obtain multiple images of the target object from different sensing device perspectives, positions, and orientations. Then, based on the multiple images of the target object from different sensing device perspectives, positions, and orientations, as well as the pixel coordinates of the images, a reference mapping table is generated to facilitate the subsequent calibration of the parameters of the sensing devices.

[0127] In one possible implementation, the above method may further include:

[0128] At preset intervals, the offset between the stabilized frame of the target image output by the sensing device and the corresponding stabilized frame of the reference image is calculated to obtain the offset of the stabilized frame of the target image; if the offset of the stabilized frame of the target image exceeds a preset offset threshold, the external parameter calibration value of the sensing device is re-determined.

[0129] In one example, the target image output by the sensing device can be monitored in real time. At preset intervals, the offset between the stabilized frame of the target image and the corresponding stabilized frame of the reference image is calculated. This offset of the target image's stabilized frame is then obtained. Further, if the offset of the target image's stabilized frame exceeds a preset offset threshold, an operation to recalibrate the external parameters of the sensing device is triggered, thereby achieving continuous calibration of the sensing device's parameters. The preset interval can be set according to actual needs; for example, it could be one day, one week, or one month.

[0130] In this embodiment of the present disclosure, the target image output by the sensing device is monitored in real time. At preset intervals, the offset between the stable frame of the target image output by the sensing device and the corresponding stable frame of the reference image is calculated. If the offset of the stable frame of the target image exceeds a preset offset threshold, the external parameter calibration value of the sensing device is re-determined to achieve continuous calibration of the parameters of the sensing device.

[0131] In one possible implementation, the above method may further include:

[0132] Perform parameter integrity verification on the redefined external parameter calibration values ​​of the sensing device, generate the parameters of the sensing device, and update the generated parameters of the sensing device into the sensing device.

[0133] For example, when the orientation of the sensing device changes, the external parameter calibration value of the sensing device is redefined to redetermine the orientation angle of the sensing device. The external parameters of the sensing device may also include the position of the sensing device, the viewing angle, etc. At this time, the parameter integrity verification of the redefined external parameter calibration value of the sensing device can be performed to generate the parameters of the sensing device containing all external parameters. The generated parameters of the sensing device are then updated in the sensing device to correct the position of the images subsequently acquired by the sensing device.

[0134] In this embodiment of the disclosure, the parameter integrity is verified on the redefined extrinsic calibration values ​​of the sensing device, parameters of the sensing device are generated, and the generated parameters of the sensing device are further updated in the sensing device to quickly realize the update of the extrinsic parameters of the sensing device, so as to promptly correct the position of the image acquired by the sensing device.

[0135] For example, taking a camera as the sensing device, such as Figure 5 As shown, Figure 5 This is a schematic diagram of an alternative image stabilization framework. The camera's single-camera sensing process decodes the acquired video data to obtain an image sequence, and then performs target detection on each image in the sequence. Figure 5After 2D and 3D detection, running field stabilization is performed, or running field stabilization is performed directly on each image in the image sequence. The running field stabilization is as described above. Figure 1 The image shown is stabilized. If it is determined that the image stabilization offset of the target frame corresponding to at least one object (which could be an obstacle as mentioned above) does not exceed a preset offset threshold, then the new position of each object is determined (i.e., ...). Figure 5 The results (from the previous step) are used to further perform 2D tracking on each object in the image, restore it to 3D before outputting the final image (i.e., the image is then output). Figure 5 (Medium-level output), which means that the output image is stabilized even when the camera moves slightly.

[0136] When the image stabilization offset of the target frame corresponding to each object in the image exceeds the preset offset threshold, the image is cached in a multi-keyframe cache queue, a displacement alarm is triggered, and the watchdog timer in the online scheduling control process is activated. Figure 5 In the WatchDog backend, a preset number of keyframe images from the cached multi-keyframe cache queue are sent to the continuous calibration process.

[0137] The continuous calibration process performs image quality diagnosis on each keyframe image in a preset number of keyframe images. Specifically, it uses the pyramid motion field method (corresponding to the pyramid feature tracking algorithm mentioned above), the feature point / line method (corresponding to the feature point matching method mentioned above), and the lane line matching method (corresponding to the lane line matching method mentioned above) to calculate the offset of the stable frame of each keyframe image from the corresponding stable frame of the reference image, thus obtaining the offset of the stable frame of each keyframe image. Then, based on the first offset of the stable frame of each keyframe image calculated using the pyramid feature tracking algorithm, the second offset of the stable frame of each keyframe image calculated using the feature point matching method, and the third offset of the stable frame of each keyframe image calculated using the lane line matching method, it is determined whether it is necessary to re-determine the camera's extrinsic parameter calibration values ​​(i.e., ...). Figure 5 (Optimal results). When it is determined that the camera's extrinsic calibration values ​​need to be redefined, a target keyframe image is selected from the multiple keyframe images. Based on the target keyframe image and a pre-determined reference mapping table, the camera's extrinsic calibration values ​​are redefined.

[0138] The watchdog timer in the online scheduling and control process starts an automatic control script to perform parameter integrity verification on the newly determined camera extrinsic parameter calibration values ​​and generate camera parameters. Figure 5The system generates parameters and uses these parameters to update the online extrinsic parameters and online ground equations of the camera during the single-camera sensing process. This allows the single-camera sensing process to correct the output image using the updated online ground equations and online extrinsic parameters before outputting the image at the object level. In other words, it enables the recalibration of camera extrinsic parameters even when the camera undergoes a large displacement.

[0139] The online scheduling and control process includes a timed task monitoring feature. This timed task monitoring can calculate the offset between the stabilized frame of the target image output by the camera and the corresponding stabilized frame of the reference image at preset intervals. The offset of the stabilized frame of the target image is obtained. If the offset of the stabilized frame of the target image exceeds a preset offset threshold, the single-camera perception process is triggered to send a preset number of keyframe images from the cached multi-keyframe cache queue to the continuous calibration process, so that the continuous calibration process can perform subsequent operations such as generating camera extrinsic parameters.

[0140] This disclosure also provides an image stabilization device, see [link to relevant documentation]. Figure 6 The device includes:

[0141] The foreground detection module 601 is used to perform foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image;

[0142] The image stabilization frame determination module 602 is used to determine the target image stabilization frame corresponding to each obstacle in each preset image stabilization frame of the current image;

[0143] The foreground occlusion determination module 603 is used to determine, based on the information of the obstacle, whether the obstacle occludes the target image frame corresponding to the obstacle;

[0144] The image stabilization offset calculation module 604 is used to calculate the offset between the target image stabilization frame corresponding to the obstacle and the corresponding image stabilization frame in the reference image when the obstacle does not obscure the target image stabilization frame corresponding to the obstacle, so as to obtain the image stabilization offset of the target image stabilization frame corresponding to the obstacle.

[0145] The position determination module 605 is used to determine the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle and the information of each obstacle, provided that the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold.

[0146] In this embodiment, foreground detection is performed on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image. Each obstacle is then traversed, and a target stable frame corresponding to each obstacle is determined within each preset stable frame of the current image. It is then determined whether each obstacle occludes its corresponding target stable frame. If not occluded, the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image is calculated to obtain the stable offset of the target stable frame corresponding to each obstacle. If the stable offset of the target stable frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the stable offset of the target stable frame corresponding to at least one obstacle and the information of each obstacle. This achieves real-time online image stabilization of the sensing device in scenarios where the sensing device undergoes small displacement, reducing the cost of manual on-site recalibration and reinstallation of the sensing device.

[0147] In one possible implementation, the above-described apparatus further includes:

[0148] The image addition module is used to add the current image to the image list when the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds a preset offset threshold.

[0149] The first extrinsic parameter calibration module is used to redetermine the extrinsic parameter calibration value of the sensing device when the number of images in the image list meets the preset extrinsic parameter calibration conditions.

[0150] In one possible implementation, the image stabilization offset calculation module 604 includes:

[0151] The first offset calculation submodule is used to calculate the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image, and obtain the target offset.

[0152] The similarity calculation submodule is used to calculate the similarity between the current image and the reference image;

[0153] The first offset determination submodule is used to determine the target offset as the image stabilization offset of the target image frame corresponding to the obstacle when the similarity meets the preset conditions, and to add the target offset to the historical offset list.

[0154] The second offset determination submodule is used to obtain each historical offset from the historical offset list when the similarity does not meet the preset conditions, and to determine the average value of each historical offset as the image stabilization offset of the target image frame corresponding to the obstacle.

[0155] In one possible implementation, the above-described apparatus further includes:

[0156] The image stabilization offset determination module is used to obtain each historical offset from the historical offset list when the obstacle occludes the target image stabilization frame corresponding to the obstacle, and determine the average value of each historical offset as the image stabilization offset of the target image stabilization frame corresponding to the obstacle.

[0157] In one possible implementation, the obstacle information is the obstacle's location information, and the location determination module 605 is specifically used for:

[0158] Calculate the homography matrix or interpolation matrix from the stable frame of the reference image to the corresponding stable frame of the current image;

[0159] Based on the location information of each obstacle, and the homography matrix or interpolation matrix, determine the new position of each obstacle.

[0160] In one possible implementation, the above-described apparatus further includes:

[0161] The displacement alarm module is used to trigger a displacement alarm for the sensing device when the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds a preset offset threshold.

[0162] In one possible implementation, the first extrinsic parameter calibration module includes:

[0163] The image acquisition submodule is used to acquire a preset number of keyframe images from the image list;

[0164] The second offset calculation submodule is used to calculate the offset between the stabilized frame of the key frame image and the corresponding stabilized frame of the reference image for each key frame image, so as to obtain the offset of the stabilized frame of the key frame image.

[0165] The extrinsic parameter calibration determination submodule is used to determine whether the extrinsic parameter calibration values ​​of the sensing device need to be redefined based on the offset of the stabilization frame of each keyframe image.

[0166] The image selection submodule is used to select a target keyframe image from keyframe images when it is determined that the external parameter calibration values ​​of the sensing device need to be redefined.

[0167] The extrinsic parameter calibration submodule is used to redetermine the extrinsic parameter calibration values ​​of the sensing device based on the target keyframe image and a pre-determined reference mapping table.

[0168] In one possible implementation, the aforementioned second offset calculation submodule is specifically used for:

[0169] Using the pyramid feature tracking algorithm, the offset of the stable frame of the keyframe image from the corresponding stable frame of the reference image is calculated, and the first offset of the stable frame of the keyframe image is obtained.

[0170] Using the feature point matching method, the offset of the stable frame of the keyframe image from the corresponding stable frame of the reference image is calculated, and the second offset of the stable frame of the keyframe image is obtained.

[0171] Using the lane line matching method, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the third offset of the stable frame of the keyframe image is obtained.

[0172] The aforementioned extrinsic parameter calibration determination submodule is specifically used to: determine whether it is necessary to re-determine the extrinsic parameter calibration values ​​of the sensing device based on the first offset of the stable frame of each key frame image calculated using the pyramid feature tracking algorithm, the second offset of the stable frame of each key frame image calculated using the feature point matching method, and the third offset of the stable frame of each key frame image calculated using the lane line matching method.

[0173] In one possible implementation, the above-described apparatus further includes:

[0174] The image acquisition module is used to capture images of the target object using multiple sensing devices, acquiring multiple images of the target object from different perspectives, positions, and orientations of the sensing devices. Each perspective, position, and orientation corresponds to one of the sensing devices in the multi-sensing system.

[0175] The mapping table generation module is used to generate a reference mapping table based on multiple images of the target object from different sensing devices, at different viewpoints, positions, and orientations, as well as the pixel coordinates of the images.

[0176] In one possible implementation, the above-described apparatus further includes:

[0177] The target offset calculation module is used to calculate the offset between the stabilized frame of the target image output by the sensing device and the corresponding stabilized frame of the reference image at preset intervals, so as to obtain the offset of the stabilized frame of the target image.

[0178] The second extrinsic parameter calibration module is used to re-determine the extrinsic parameter calibration value of the sensing device when the offset of the stable frame of the target image exceeds a preset offset threshold.

[0179] In one possible implementation, the above-described apparatus further includes:

[0180] The parameter generation module is used to verify the integrity of the redefined external parameter calibration values ​​of the sensing device and generate the parameters of the sensing device.

[0181] The parameter update module is used to update the generated parameters of the sensing device into the sensing device.

[0182] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals. It should be noted that the head model in this embodiment is not a head model specific to any particular user and does not reflect the personal information of any particular user. It should also be noted that the two-dimensional face images in this embodiment are from publicly available datasets.

[0183] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0184] This disclosure provides an electronic device, comprising:

[0185] At least one processor; and

[0186] A memory that is communicatively connected to at least one processor; wherein,

[0187] The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform any of the methods of this disclosure.

[0188] This disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform any of the methods described in this disclosure.

[0189] This disclosure provides a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in this disclosure.

[0190] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0191] like Figure 7As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0192] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0193] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as image stabilization methods. For example, in some embodiments, the image stabilization method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the image stabilization method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the image stabilization method by any other suitable means (e.g., by means of firmware).

[0194] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0195] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0196] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0197] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0198] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0199] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0200] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image stabilization method, comprising: Foreground detection is performed on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image; For each obstacle, a target image stabilization frame corresponding to the obstacle is determined in each preset image stabilization frame of the current image; Based on the information about the obstacle, determine whether the obstacle occludes the target frame corresponding to the obstacle; If the obstacle does not obscure the target frame corresponding to the obstacle, calculate the offset between the target frame corresponding to the obstacle and the corresponding frame in the reference image to obtain the image stabilization offset of the target frame corresponding to the obstacle. If the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold, the new position of each obstacle is determined based on the image stabilization offset of the target image stabilization frame corresponding to the at least one obstacle and the information of each obstacle. If the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold, the current image is added to the image list. If the number of images in the image list meets the preset extrinsic calibration conditions, the extrinsic calibration value of the sensing device is redefined; The process of re-determining the extrinsic calibration values ​​of the sensing device includes: acquiring a preset number of keyframe images from the image list; for each keyframe image, calculating the offset between the stabilized frame of the keyframe image and the corresponding stabilized frame of the reference image to obtain the offset of the stabilized frame of the keyframe image; based on the offset of the stabilized frame of each keyframe image, determining whether it is necessary to re-determine the extrinsic calibration values ​​of the sensing device; when it is determined that it is necessary to re-determine the extrinsic calibration values ​​of the sensing device, selecting a target keyframe image from the keyframe images; and based on the target keyframe image and a pre-determined reference mapping table, re-determining the extrinsic calibration values ​​of the sensing device. The reference mapping table is obtained through the following steps: multiple images of the target object are captured by multiple sensing devices from different perspectives, positions, and orientations of the target object, wherein each perspective, position, and orientation of the sensing device corresponds to one of the multiple sensing devices; the reference mapping table is generated based on the multiple images of the target object from different perspectives, positions, and orientations of the sensing devices, and the pixel coordinates of the images.

2. The method according to claim 1, wherein calculating the offset between the target stabilized frame corresponding to the obstacle and the corresponding stabilized frame in the reference image to obtain the stabilized offset of the target stabilized frame corresponding to the obstacle includes: Calculate the offset between the target frame corresponding to the obstacle and the corresponding frame in the reference image to obtain the target offset; Calculate the similarity between the current image and the reference image; If the similarity meets the preset conditions, the target offset is determined as the image stabilization offset of the target image frame corresponding to the obstacle, and the target offset is added to the historical offset list. If the similarity does not meet the preset conditions, each historical offset is obtained from the historical offset list, and the average value of each historical offset is determined as the image stabilization offset of the target image frame corresponding to the obstacle.

3. The method according to claim 1, further comprising: When an obstacle obscures the target frame corresponding to the obstacle, each historical offset is obtained from the historical offset list, and the average of each historical offset is determined as the image stabilization offset of the target frame corresponding to the obstacle.

4. The method according to claim 1, wherein the information of the obstacle is the position information of the obstacle, and the step of determining the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to the at least one obstacle and the information of each obstacle includes: Calculate the homography matrix or interpolation matrix from the stable frame of the reference image to the corresponding stable frame of the current image; Based on the location information of each obstacle, and the homography matrix or the interpolation matrix, the new position of each obstacle is determined.

5. The method according to claim 1, further comprising: If the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold, a displacement alarm of the sensing device will be triggered.

6. The method according to claim 1, wherein calculating the offset of the stabilized frame of the keyframe image from the corresponding stabilized frame of the reference image to obtain the offset of the stabilized frame of the keyframe image comprises: Using the pyramid feature tracking algorithm, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the first offset of the stable frame of the keyframe image is obtained. Using the feature point matching method, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the second offset of the stable frame of the keyframe image is obtained. Using the lane line matching method, the offset between the stabilized frame of the keyframe image and the corresponding stabilized frame of the reference image is calculated to obtain the third offset of the stabilized frame of the keyframe image. The determination of whether the extrinsic parameter calibration values ​​of the sensing device need to be re-determined based on the offset of the image stabilization frame of each keyframe image includes: Based on the first offset of the stable frame of each key frame image calculated using the pyramid feature tracking algorithm, the second offset of the stable frame of each key frame image calculated using the feature point matching method, and the third offset of the stable frame of each key frame image calculated using the lane line matching method, it is determined whether the external parameter calibration value of the sensing device needs to be re-determined.

7. The method according to claim 1, further comprising: At preset intervals, the offset between the stabilized frame of the target image output by the sensing device and the corresponding stabilized frame of the reference image is calculated to obtain the offset of the stabilized frame of the target image. If the offset of the stabilization frame of the target image exceeds the preset offset threshold, the extrinsic parameter calibration value of the sensing device is re-determined.

8. The method according to any one of claims 1-7, further comprising: Perform parameter integrity verification on the redefined external parameter calibration values ​​of the sensing device, and generate the parameters of the sensing device; Update the generated parameters of the sensing device to the sensing device.

9. An image stabilization device, comprising: The foreground detection module is used to perform foreground detection on the current image acquired by the sensing device to obtain information about each obstacle contained in the current image; The image stabilization frame determination module is used to determine the target image stabilization frame corresponding to each obstacle in each preset image stabilization frame of the current image; The foreground occlusion determination module is used to determine whether the obstacle occludes the target frame corresponding to the obstacle based on the information of the obstacle; The image stabilization offset calculation module is used to calculate the offset between the target image stabilization frame corresponding to the obstacle and the corresponding image stabilization frame in the reference image when the obstacle does not obscure the target image stabilization frame corresponding to the obstacle, and obtain the image stabilization offset of the target image stabilization frame corresponding to the obstacle. The position determination module is used to determine the new position of each obstacle based on the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle and the information of each obstacle, when the image stabilization offset of the target image stabilization frame corresponding to at least one obstacle does not exceed a preset offset threshold. The image addition module is used to add the current image to the image list when the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold. The first extrinsic parameter calibration module is used to re-determine the extrinsic parameter calibration value of the sensing device when the number of images in the image list meets the preset extrinsic parameter calibration conditions. The first extrinsic parameter calibration module includes: The image acquisition submodule is used to acquire a preset number of keyframe images from the image list; The second offset calculation submodule is used to calculate the offset between the stabilized frame of the key frame image and the corresponding stabilized frame of the reference image for each key frame image, so as to obtain the offset of the stabilized frame of the key frame image. The extrinsic parameter calibration determination submodule is used to determine whether the extrinsic parameter calibration values ​​of the sensing device need to be re-determined based on the offset of the image stabilization frame of each key frame image. The image selection submodule is used to select a target keyframe image from the keyframe images when it is determined that the external parameter calibration value of the sensing device needs to be redefined. The extrinsic parameter calibration submodule is used to re-determine the extrinsic parameter calibration values ​​of the sensing device based on the target keyframe image and a pre-determined reference mapping table. The device further includes: The image acquisition module is used to capture images of the target object using multiple sensing devices, acquiring multiple images of the target object from different perspectives, positions, and orientations of the sensing devices, wherein each perspective, position, and orientation of the sensing device corresponds to one of the multiple sensing devices; The mapping table generation module is used to generate the reference mapping table based on multiple images of the target object from different sensing device perspectives, positions, and orientations, as well as the pixel coordinates of the images.

10. The apparatus according to claim 9, wherein the image stabilization offset calculation module comprises: The first offset calculation submodule is used to calculate the offset between the target stable frame corresponding to the obstacle and the corresponding stable frame in the reference image, and obtain the target offset. A similarity calculation submodule is used to calculate the similarity between the current image and the reference image; The first offset determination submodule is used to determine the target offset as the image stabilization offset of the target image stabilization frame corresponding to the obstacle when the similarity meets the preset conditions, and to add the target offset to the historical offset list. The second offset determination submodule is used to obtain each historical offset from the historical offset list when the similarity does not meet the preset conditions, and determine the average value of each historical offset as the image stabilization offset of the target image stabilization frame corresponding to the obstacle.

11. The apparatus according to claim 9, further comprising: The image stabilization offset determination module is used to obtain each historical offset from the historical offset list when the obstacle occludes the target image stabilization frame corresponding to the obstacle, and determine the average value of each historical offset as the image stabilization offset of the target image stabilization frame corresponding to the obstacle.

12. The apparatus according to claim 9, wherein the information of the obstacle is the position information of the obstacle, and the position determination module is specifically used for: Calculate the homography matrix or interpolation matrix from the stable frame of the reference image to the corresponding stable frame of the current image; Based on the location information of each obstacle, and the homography matrix or the interpolation matrix, the new position of each obstacle is determined.

13. The apparatus according to claim 9, further comprising: The displacement alarm module is used to trigger a displacement alarm for the sensing device when the image stabilization offset of the target image stabilization frame corresponding to each obstacle in the current image exceeds the preset offset threshold.

14. The apparatus according to claim 9, wherein the second offset calculation submodule is specifically used for: Using the pyramid feature tracking algorithm, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the first offset of the stable frame of the keyframe image is obtained. Using the feature point matching method, the offset between the stable frame of the keyframe image and the corresponding stable frame of the reference image is calculated, and the second offset of the stable frame of the keyframe image is obtained. Using the lane line matching method, the offset between the stabilized frame of the keyframe image and the corresponding stabilized frame of the reference image is calculated to obtain the third offset of the stabilized frame of the keyframe image. The extrinsic parameter calibration determination submodule is specifically used to: determine whether it is necessary to re-determine the extrinsic parameter calibration value of the sensing device based on the first offset of the stable frame of each key frame image calculated using the pyramid feature tracking algorithm, the second offset of the stable frame of each key frame image calculated using the feature point matching method, and the third offset of the stable frame of each key frame image calculated using the lane line matching method.

15. The apparatus according to claim 9, further comprising: The target offset calculation module is used to calculate the offset between the stabilized frame of the target image output by the sensing device and the corresponding stabilized frame of the reference image at preset time intervals, so as to obtain the offset of the stabilized frame of the target image. The second extrinsic parameter calibration module is used to re-determine the extrinsic parameter calibration value of the sensing device when the offset of the stabilization frame of the target image exceeds the preset offset threshold.

16. The apparatus according to any one of claims 9-15, further comprising: The parameter generation module is used to perform parameter integrity verification on the redefined external parameter calibration values ​​of the sensing device and generate the parameters of the sensing device. The parameter update module is used to update the generated parameters of the sensing device into the sensing device.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images