Pose Calibration Method, Device, Storage Medium and Terminal Device of Image Sensor

Abstract

The present invention discloses a method, device, storage medium and terminal device for pose calibration of an image sensor, including: obtaining a background image within the monitoring field of view of the image sensor to be calibrated; obtaining N motion images when there are moving objects within the monitoring field of view, and respectively obtaining the position information of the moving object corresponding to each motion image in the world coordinate system; where N>0; calculating N mask images based on the background image and the N motion images; calculating the pose parameters of the image sensor to be calibrated based on the N position information corresponding to the moving object, the N mask images and the internal parameter matrix of the image sensor to be calibrated. The technical solution of the present invention can improve the accuracy of pose calibration of the image sensor.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor calibration, and particularly to a method, device, computer-readable storage medium and terminal device for pose calibration of an image sensor. Background Art

[0002] Depth sensors are currently popular sensors. They can use the ToF (Time of Flight) or structured light method to obtain the depth values of the scene within the field of view, that is, the distance from the sensor. The data obtained by the depth sensor is a two-dimensional array, and each position in the array stores the depth value. Visualizing this two-dimensional array as a picture gives the depth map.

[0003] The role of calibration is to obtain the parameters of the depth sensor. The parameters of the depth sensor are divided into internal parameters and external parameters. Among them, the internal parameters refer to the self-parameters of the depth sensor, which are generally given by the manufacturer or calibrated in advance. The external parameters refer to the pose of the depth sensor relative to the world coordinate system. The external parameters determine the relative position relationship between the coordinates corresponding to the depth sensor and the world coordinate system. In practical applications, the three-dimensional distance of the detected target can be obtained through the depth sensor, and the original data measured by the depth sensor is based on the data in its own coordinate system. It is necessary to convert the measured original data to the world coordinate system through calibration and then perform subsequent processing. Therefore, the accuracy of calibration plays an important role in the use of the depth sensor.

[0004] Currently, in the process of external parameter calibration of depth sensors, the existing calibration methods often adjust the test position manually. However, different people adjusting the test position may lead to different errors in the test position, thus affecting the calibration accuracy, and the errors in the test position may also cause inaccurate calibration. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a method, device, computer-readable storage medium and terminal device for pose calibration of an image sensor, which can improve the accuracy of pose calibration of the image sensor.

[0006] To solve the above technical problem, the embodiments of the present invention provide a method for pose calibration of an image sensor, including:

[0007] Obtain a background image within the monitoring field of view of the image sensor to be calibrated;

[0008] Obtain N motion images when there are moving objects within the monitoring field of view, and respectively obtain the position information of the moving objects corresponding to each motion image in the world coordinate system; where N>0;

[0009] Calculate and obtain N mask images based on the background image and the N motion images;

[0010] Calculate and obtain the pose parameters of the image sensor to be calibrated based on the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated.

[0011] Further, the obtaining of the background image within the monitoring field of view of the image sensor to be calibrated specifically includes:

[0012] Obtain the background depth image when there is no moving object within the monitoring field of view;

[0013] Perform background modeling based on the background depth image to obtain the background image.

[0014] Further, the method obtains the position information of the moving object corresponding to each motion image in the world coordinate system through the following steps:

[0015] Use the SLAM algorithm to map the environment of the monitoring area to obtain an environmental grid map;

[0016] Obtain the position information of the moving object corresponding to each motion image in the world coordinate system based on the environmental grid map.

[0017] Further, the calculating and obtaining of the N mask images based on the background image and the N motion images specifically includes:

[0018] Perform frame difference calculation on the background image and each motion image respectively to obtain N frame difference images correspondingly;

[0019] Process each frame difference image according to a preset depth threshold to obtain the N mask images correspondingly.

[0020] Further, the method obtains the i-th mask image through the following steps:

[0021] Compare the depth value of each pixel point on the i-th frame difference image with the depth threshold respectively; where i = 1, 2,..., N;

[0022] When the depth value of any pixel point is greater than the depth threshold, mark this pixel point as 1;

[0023] When the depth value of any pixel point is not greater than the depth threshold, mark this pixel point as 0;

[0024] Obtain the i-th mask image correspondingly according to the i-th frame difference image after the marking process.

[0025] Further, calculating the pose parameters of the image sensor to be calibrated according to the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated specifically includes:

[0026] Calculating the average value of the coordinate values and depth values of all pixels marked as 1 on each mask image respectively to obtain N clustering centers correspondingly; where the clustering center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, and d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2,..., N;

[0027] Calculating the pose parameters of the image sensor to be calibrated according to the N position information, the N clustering centers, and the internal parameter matrix of the image sensor to be calibrated.

[0028] Further, calculating the pose parameters of the image sensor to be calibrated according to the N position information, the N clustering centers, and the internal parameter matrix of the image sensor to be calibrated specifically includes:

[0029] Solving according to the formula to correspondingly obtain the pose parameter H d of the image sensor to be calibrated; where P i represents the i-th position information corresponding to the moving object, represents the position information obtained by converting the i-th clustering center p i =(u i , v i , d i ) to the world coordinate system, and K s represents the internal parameter matrix of the image sensor to be calibrated.

[0030] To solve the above technical problems, an embodiment of the present invention further provides a pose calibration device for an image sensor, including:

[0031] A background image acquisition module, configured to acquire a background image within the monitoring field of view of the image sensor to be calibrated;

[0032] A moving image acquisition module, configured to acquire N moving images when there is a moving object in the monitoring field of view, and respectively acquire the position information of the moving object corresponding to each moving image in the world coordinate system; where N>0;

[0033] A mask image acquisition module, configured to calculate and obtain N mask images according to the background image and the N moving images;

[0034] A sensor pose acquisition module, configured to calculate and obtain the pose parameters of the to-be-calibrated image sensor according to N position information corresponding to the moving body, the N mask images, and the internal parameter matrix of the to-be-calibrated image sensor.

[0035] An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the pose calibration method of the image sensor described in any one of the above.

[0036] An embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the pose calibration method of the image sensor described in any one of the above.

[0037] Compared with the prior art, an embodiment of the present invention provides a pose calibration method, device, computer-readable storage medium, and terminal device for an image sensor. By acquiring a background image within the monitoring field of view of the to-be-calibrated image sensor, acquiring N moving images when there is a moving body within the monitoring field of view, and respectively acquiring the position information of the moving body corresponding to each moving image in the world coordinate system, N mask images are calculated and obtained according to the background image and the N moving images, and the pose parameters of the to-be-calibrated image sensor are calculated and obtained according to N position information corresponding to the moving body, the N mask images, and the internal parameter matrix of the to-be-calibrated image sensor, which can greatly reduce the error caused by manual adjustment of the test position, thereby improving the accuracy of the pose calibration of the image sensor. Description of the Drawings

[0038] Figure 1 is a flowchart of a preferred embodiment of a pose calibration method for an image sensor provided by the present invention;

[0039] Figure 2 is a structural block diagram of a preferred embodiment of a pose calibration device for an image sensor provided by the present invention;

[0040] Figure 3 is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Embodiments

[0041] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the technical field of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0042] An embodiment of the present invention provides a method for pose calibration of an image sensor. Refer to Figure 1 As shown, it is a flowchart of a preferred embodiment of a method for pose calibration of an image sensor provided by the present invention. The method includes steps S11 to S14:

[0043] Step S11: Obtain a background image within the monitoring field of view of the image sensor to be calibrated;

[0044] Step S12: Obtain N motion images when there are moving objects in the monitoring field of view, and respectively obtain the position information of the moving object corresponding to each motion image in the world coordinate system; where N>0;

[0045] Step S13: Calculate and obtain N mask images based on the background image and the N motion images;

[0046] Step S14: Calculate and obtain the pose parameters of the image sensor to be calibrated based on the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated.

[0047] Specifically, first, when there is no moving object (i.e., no motion) within the monitoring field of view of the image sensor to be calibrated, a background image within the monitoring field of view is collected and obtained through the image sensor to be calibrated. When there is a moving object (i.e., there is motion) within the monitoring field of view of the image sensor to be calibrated, N motion images corresponding to the moving object at N (N>0) different positions are collected and obtained through the image sensor to be calibrated, and the position information of the moving object corresponding to each motion image in the world coordinate system is respectively obtained, and N position information corresponding to the moving object is correspondingly obtained; then, the obtained background image is respectively calculated with each motion image, and N mask images are correspondingly obtained; finally, based on the N position information corresponding to the obtained moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated, the pose parameters of the image sensor to be calibrated are calculated.

[0048] It should be noted that the mobile object can be a mobile robot. By controlling the movement of the mobile robot and simultaneously performing motion detection on the depth image collected by the image sensor to be calibrated, it can be determined whether there is a mobile robot within the monitoring field of view of the image sensor to be calibrated. It can be understood that when it is determined that no motion can be detected within the monitoring field of view of the image sensor to be calibrated (that is, the mobile robot is not within the monitoring field of view, or not on the depth image), the corresponding depth image collected is the background depth image. When it is determined that motion is detected within the monitoring field of view of the image sensor to be calibrated (that is, the mobile robot is within the monitoring field of view, or on the depth image), the corresponding depth image collected is the motion depth image.

[0049] Among them, it is possible to determine whether there is a mobile robot within the monitoring field of view of the image sensor to be calibrated according to the mask image corresponding to the depth image. For example, when the number of pixel points marked as 1 in the mask image is less than a certain threshold, it is determined that there is no motion, that is, there is no mobile robot within the monitoring field of view of the image sensor to be calibrated. Otherwise, it is determined that there is motion, that is, there is a mobile robot within the monitoring field of view of the image sensor to be calibrated.

[0050] A method for pose calibration of an image sensor provided by an embodiment of the present invention, by obtaining a background image and N motion images within the monitoring field of view of the image sensor to be calibrated, and respectively obtaining the position information of the mobile object corresponding to each motion image in the world coordinate system, so as to calculate and obtain N mask images according to the background image and the N motion images, and calculate and obtain the pose parameters of the image sensor to be calibrated according to the N position information corresponding to the mobile object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated, can greatly reduce the error caused by manual adjustment of the test position, thereby improving the accuracy of the pose calibration of the image sensor.

[0051] In another preferred embodiment, the obtaining of the background image within the monitoring field of view of the image sensor to be calibrated specifically includes:

[0052] Obtain the background depth image when there is no mobile object within the monitoring field of view;

[0053] Perform background modeling according to the background depth image to obtain the background image.

[0054] Specifically, in combination with the above embodiments, when actually obtaining the background image within the monitoring field of view of the image sensor to be calibrated, at least one background depth image when there is no mobile object within the monitoring field of view can be collected first by the image sensor to be calibrated, and then background modeling is performed according to the at least one collected background depth image, and the background image within the monitoring field of view of the image sensor to be calibrated is correspondingly obtained.

[0055] It should be noted that during the calibration process, the same image sensor to be calibrated generally does not rotate, that is, the pose remains unchanged. In this case, at least one background depth image collected is similar. Due to the existence of noise, there will be relatively small differences in each background depth image. Therefore, when obtaining the background image within the monitoring field of view of the image sensor to be calibrated based on at least one background depth image collected, one background depth image can be used to correspondingly obtain the background image within the monitoring field of view of the image sensor to be calibrated, or multiple background depth images can be used to correspondingly obtain the background image within the monitoring field of view of the image sensor to be calibrated; among them, if multiple background depth images are used, the depth values of each pixel point in the multiple background depth images can be averaged to correspondingly obtain the background image within the monitoring field of view of the image sensor to be calibrated.

[0056] In yet another preferred embodiment, the method obtains the position information of the moving object corresponding to each motion image in the world coordinate system through the following steps:

[0057] Use the SLAM algorithm to map the environment of the monitoring area to obtain an environmental grid map;

[0058] Obtain the position information of the moving object corresponding to each motion image in the world coordinate system according to the environmental grid map.

[0059] Specifically, in combination with the above embodiments, when obtaining the position information of the moving object corresponding to each motion image in the world coordinate system, a mobile robot (i.e., the moving object) and the SLAM algorithm (Simultaneous Localization and Mapping) can be used to map the environment of the monitoring area of the image sensor to be calibrated, and the corresponding environmental grid map of the monitoring area can be obtained to determine the world coordinate system. The mobile robot can determine its own position information in the world coordinate system (i.e., the three-dimensional space coordinate value in the world coordinate system) according to the constructed environmental grid map and the SLAM positioning function, so that the position information of the moving object corresponding to each motion image in the world coordinate system can be correspondingly obtained.

[0060] In yet another preferred embodiment, the calculating the N mask images according to the background image and the N motion images specifically includes:

[0061] Perform frame difference calculations on the background image and each motion image respectively to correspondingly obtain N frame difference images;

[0062] Process each frame difference image according to a preset depth threshold to correspondingly obtain the N mask images.

[0063] Specifically, in combination with the above embodiments, after obtaining the background image and N motion images within the monitoring field of view of the image sensor to be calibrated, the obtained background image is respectively subjected to frame difference calculation with each motion image, and N frame difference images are correspondingly obtained. Then, according to the preset depth threshold, each frame difference image is correspondingly processed, and N mask images are correspondingly obtained.

[0064] As an improvement of the above solution, the method obtains the i-th mask image through the following steps:

[0065] Compare the depth value of each pixel point on the i-th frame difference image with the depth threshold respectively; where i = 1, 2,..., N;

[0066] When the depth value of any pixel point is greater than the depth threshold, mark this pixel point as 1;

[0067] When the depth value of any pixel point is not greater than the depth threshold, mark this pixel point as 0;

[0068] Obtain the i-th mask image according to the i-th frame difference image after the marking process.

[0069] Specifically, in combination with the above embodiments, the processing method of each frame difference image is the same. Here, the processing method of the i-th frame difference image is taken as an example for illustration. Compare the depth value of each pixel point on the i-th frame difference image with the preset depth threshold respectively. When it is determined that the depth value of any pixel point is greater than the preset depth threshold, mark this pixel point as 1. When it is determined that the depth value of any pixel point is not greater than the preset depth threshold, mark this pixel point as 0; correspondingly, after marking each pixel point on the i-th frame difference image as 1 or 0, obtain the i-th mask image according to the i-th frame difference image after the marking process.

[0070] It should be noted that when collecting motion images through the image sensor to be calibrated, due to the certain thickness of the moving body, the positions of the pixel points marked as 1 on the mask image are exactly caused by the moving body. When the moving body moves to this position, the depth of the pixel points at this position will change accordingly.

[0071] In yet another preferred embodiment, calculating the pose parameters of the image sensor to be calibrated according to the N position information corresponding to the moving body, the N mask images, and the internal parameter matrix of the image sensor to be calibrated specifically includes:

[0072] Calculate the average value of the coordinate values and depth values of all pixel points marked as 1 on each mask image respectively, and correspondingly obtain N cluster centers; where the cluster center corresponding to the i-th mask image is p i =(ui , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, and d i represents the average depth value of all pixels marked as 1 on the i-th mask image, where i = 1, 2,..., N;

[0073] Based on the N position information, the N cluster centers, and the internal parameter matrix of the image sensor to be calibrated, the pose parameters of the image sensor to be calibrated are calculated and obtained.

[0074] Specifically, in combination with the above embodiments, when calculating and obtaining the pose parameters of the image sensor to be calibrated based on the N position information, N mask images, and the internal parameter matrix of the image sensor to be calibrated corresponding to the moving body, it is necessary to further process the obtained N mask images, that is, calculate the average values of the coordinate values and depth values of all pixels marked as 1 on each mask image respectively, corresponding to obtain N cluster centers, and then calculate and obtain the pose parameters of the image sensor to be calibrated based on the N position information, N cluster centers, and the internal parameter matrix of the image sensor to be calibrated corresponding to the moving body.

[0075] Among them, the processing method of each mask image is the same. Here, taking the processing method of the i-th mask image as an example, the average values of the uv coordinate values and depth value d of all pixels marked as 1 on the i-th mask image are calculated, and the corresponding cluster center of the i-th mask image is obtained and represented as p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, and d i represents the average depth value of all pixels marked as 1 on the i-th mask image, where i = 1, 2,..., N.

[0076] It should be noted that the internal parameter matrix of the image sensor to be calibrated is generally given by the manufacturer or pre-calibrated and belongs to known parameters.

[0077] As an improvement of the above solution, the calculating and obtaining the pose parameters of the image sensor to be calibrated based on the N position information, the N cluster centers, and the internal parameter matrix of the image sensor to be calibrated specifically includes:

[0078] Solve according to the formula to correspondingly obtain the pose parameter H of the image sensor to be calibratedd ; wherein, P i represents the i-th position information corresponding to the moving body, represents converting the i-th clustering center p i =(u i , v i , d i ) to the position information in the world coordinate system, and K s represents the internal parameter matrix of the image sensor to be calibrated.

[0079] Specifically, in combination with the above embodiments, when calculating the pose parameters of the image sensor to be calibrated based on the N position information, N clustering centers corresponding to the moving body, and the internal parameter matrix of the image sensor to be calibrated, the formula can be used for solution, and the pose parameters H d of the image sensor to be calibrated are obtained accordingly. P i represents the i-th position information (three-dimensional space coordinate value) corresponding to the moving body, represents converting the i-th clustering center p i =(u i , v i , d i ) to the position information (three-dimensional space coordinate value) in the world coordinate system, K s represents the internal parameter matrix of the image sensor to be calibrated, represents the Euclidean distance between two three-dimensional vectors.

[0080] It should be noted that the pose parameter H d of the image sensor to be calibrated is the spatial transformation matrix of the image sensor to be calibrated or the pose of the image sensor to be calibrated relative to the world coordinate system. Through the pose parameter H d , a point (u, v) on the depth image collected by the image sensor to be calibrated can be converted into a three-dimensional coordinate value (x, y, z) in the world coordinate system.

[0081] The embodiment of the present invention also provides a pose calibration device for an image sensor. Refer to Figure 2 shown, which is a structural block diagram of a preferred embodiment of a pose calibration device for an image sensor provided by the present invention. The device includes:

[0082] A background image acquisition module 11, configured to acquire a background image within the monitoring field of view of the image sensor to be calibrated;

[0083] A motion image acquisition module 12, configured to acquire N motion images when there is a moving body within the monitoring field of view, and respectively acquire the position information of the moving body corresponding to each motion image in the world coordinate system; wherein, N>0;

[0084] A mask image acquisition module 13, configured to calculate and obtain N mask images according to the background image and the N moving images;

[0085] A sensor pose acquisition module 14, configured to calculate and obtain the pose parameters of the to-be-calibrated image sensor according to N position information corresponding to the moving body, the N mask images, and the internal parameter matrix of the to-be-calibrated image sensor.

[0086] Preferably, the background image acquisition module 11 specifically includes:

[0087] A background depth image acquisition unit, configured to acquire a background depth image when there is no moving body in the monitoring field of view;

[0088] A background image acquisition unit, configured to perform background modeling according to the background depth image to obtain the background image.

[0089] Preferably, the moving image acquisition module 12 specifically includes a position information acquisition unit, configured to:

[0090] Build a map of the monitoring area environment by using the SLAM algorithm to obtain an environmental grid map;

[0091] Obtain the position information of the moving body corresponding to each moving image in the world coordinate system according to the environmental grid map.

[0092] Preferably, the mask image acquisition module 13 specifically includes:

[0093] A frame difference image calculation unit, configured to perform frame difference calculation on the background image and each moving image respectively to obtain N frame difference images correspondingly;

[0094] A mask image acquisition unit, configured to process each frame difference image according to a preset depth threshold to obtain the N mask images correspondingly.

[0095] Preferably, the mask image acquisition unit is specifically configured to:

[0096] Compare the depth value of each pixel point on the i-th frame difference image with the depth threshold respectively; where i = 1, 2,..., N;

[0097] When the depth value of any pixel point is greater than the depth threshold, mark the pixel point as 1;

[0098] When the depth value of any pixel point is not greater than the depth threshold, mark the pixel point as 0;

[0099] Obtain the i-th mask image correspondingly according to the i-th frame difference image after the marking process.

[0100] Preferably, the sensor pose acquisition module 14 specifically includes:

[0101] A mask image processing unit, configured to calculate the average value of the coordinate values and depth values of all pixels marked as 1 on each mask image, and correspondingly obtain N clustering centers; wherein, the clustering center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, and d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2,..., N;

[0102] A sensor pose acquisition unit, configured to calculate and obtain the pose parameters of the image sensor to be calibrated according to the N position information, the N clustering centers, and the internal parameter matrix of the image sensor to be calibrated.

[0103] Preferably, the sensor pose acquisition unit is specifically configured to:

[0104] Solve according to the formula to correspondingly obtain the pose parameter H d of the image sensor to be calibrated; wherein, P i represents the i-th position information corresponding to the moving body, represents the position information obtained by converting the i-th clustering center p i =(u i , v i , d i ) to the world coordinate system, and K s represents the internal parameter matrix of the image sensor to be calibrated.

[0105] It should be noted that the pose calibration device for an image sensor provided in the embodiments of the present invention can implement all the processes of the pose calibration method for an image sensor described in any of the above embodiments. The functions and achieved technical effects of each module and unit in the device are respectively the same as those of the pose calibration method for an image sensor described in the above embodiments, and will not be elaborated here.

[0106] The embodiments of the present invention further provide a computer-readable storage medium, and the computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the pose calibration method for an image sensor described in any of the above embodiments.

[0107] An embodiment of the present invention further provides a terminal device. Refer to Figure 3 As shown in Figure 3 , it is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10. When the processor 10 executes the computer program, it implements the pose calibration method of the image sensor described in any of the above embodiments.

[0108] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2,...). The one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device.

[0109] The processor 10 can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 10 can also be any conventional processor. The processor 10 is the control center of the terminal device, and connects various parts of the terminal device through various interfaces and lines.

[0110] The memory 20 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc., and the data storage area can store relevant data, etc. In addition, the memory 20 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., or the memory 20 can also be other volatile solid-state storage devices.

[0111] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 3The structural block diagram is only an example of the above terminal device, and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine some components, or different components.

[0112] In summary, for a pose calibration method, device, computer-readable storage medium, and terminal device of an image sensor provided by an embodiment of the present invention, by acquiring a background image and N moving images within the monitoring field of view of the image sensor to be calibrated, and respectively acquiring the position information of the moving object corresponding to each moving image in the world coordinate system, N mask images are calculated based on the background image and the N moving images, and the pose parameters of the image sensor to be calibrated are calculated based on the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated, which can greatly reduce the error caused by manual adjustment of the test position, thereby improving the accuracy of the pose calibration of the image sensor.

[0113] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A method for pose calibration of an image sensor, characterized in that, Including: Obtain a background image within the monitoring field of view of the image sensor to be calibrated; Obtain N motion images when there are moving objects within the monitoring field of view, and respectively obtain the position information of the moving object corresponding to each motion image in the world coordinate system; where N>0; Calculate and obtain N mask images based on the background image and the N motion images; Calculate and obtain the pose parameters of the image sensor to be calibrated based on the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated; The calculating and obtaining the pose parameters of the image sensor to be calibrated based on the N position information corresponding to the moving object, the N mask images, and the internal parameter matrix of the image sensor to be calibrated specifically includes: Calculate the average value of the coordinate values and depth values of all pixels marked as 1 on each mask image, and obtain N cluster centers correspondingly; among them, the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2,..., N; Calculate and obtain the pose parameters of the image sensor to be calibrated based on the N position information, the N cluster centers, and the internal parameter matrix of the image sensor to be calibrated.

2. The pose calibration method of the image sensor according to claim 1, characterized in that The obtaining a background image within the monitoring field of view of the image sensor to be calibrated specifically includes: Obtain a background depth image when there are no moving objects within the monitoring field of view; Perform background modeling based on the background depth image to obtain the background image.

3. The pose calibration method of the image sensor according to claim 1, characterized in that, The method obtains the position information of the moving object corresponding to each motion image in the world coordinate system through the following steps: Use the SLAM algorithm to build a map of the monitoring area environment to obtain an environmental grid map; Obtain the position information of the moving object corresponding to each motion image in the world coordinate system based on the environmental grid map.

4. The pose calibration method of the image sensor according to claim 1, characterized in that The calculating and obtaining N mask images based on the background image and the N motion images specifically includes: Perform frame difference calculation on the background image and each motion image respectively to obtain N frame difference images correspondingly; Process each frame difference image according to a preset depth threshold to obtain the N mask images correspondingly.

5. The pose calibration method of the image sensor according to claim 4, wherein, The method obtains the i-th mask image through the following steps: Compare the depth value of each pixel point on the i-th frame difference image with the depth threshold respectively; where i = 1, 2,..., N; When the depth value of any pixel point is greater than the depth threshold, mark this pixel point as 1; When the depth value of any pixel point is not greater than the depth threshold, mark this pixel point as 0; Obtain the i-th mask image corresponding to the i-th frame difference image after the marking process.

6. The pose calibration method of the image sensor according to any one of claims 1, characterized in that The calculating and obtaining the pose parameters of the image sensor to be calibrated based on the N position information, the N cluster centers, and the internal parameter matrix of the image sensor to be calibrated specifically includes: Solve according to the formula to obtain the pose parameter H of the image sensor to be calibrated accordingly d ; where P i represents the i-th position information corresponding to the moving body, represents converting the i-th cluster center p i =(u i , v i , d i ) to the position information in the world coordinate system, and K s represents the internal parameter matrix of the image sensor to be calibrated.

7. An attitude calibration device for an image sensor, characterized in that Including: A background image acquisition module for obtaining a background image within the monitoring field of view of the image sensor to be calibrated; A motion image acquisition module for obtaining N motion images when there are moving objects within the monitoring field of view, and respectively obtaining the position information of the moving object corresponding to each motion image in the world coordinate system; where N>0; A mask image acquisition module for calculating and obtaining N mask images based on the background image and the N motion images; The sensor pose acquisition module is used to calculate and obtain the pose parameters of the image sensor to be calibrated according to the N position information corresponding to the mobile body, the N mask images, and the internal parameter matrix of the image sensor to be calibrated; Specifically, the sensor pose acquisition module includes: A mask image processing unit is used to calculate the average values of the coordinate values and depth values of all the pixels marked as 1 on each mask image respectively, and correspondingly obtain N clustering centers; among them, the clustering center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all the pixels marked as 1 on the i-th mask image, and d i represents the average depth value of all the pixels marked as 1 on the i-th mask image, i = 1, 2,..., N; The sensor pose acquisition unit is used to calculate and obtain the pose parameters of the image sensor to be calibrated according to the N position information, the N clustering centers, and the internal parameter matrix of the image sensor to be calibrated.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the pose calibration method of the image sensor according to any one of claims 1 to 6.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the pose calibration method of the image sensor according to any one of claims 1 to 6.

Images