A method for recalibrating a camera perception system and a camera perception system

Abstract

The present invention provides a recalibration method for a camera perception system, which is used to calibrate a current virtual world coordinate system used by the camera perception system based on a posture change of the camera relative to an initial posture, wherein after the first calibration, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, including: S1, obtaining the posture change of the camera based on a tilt sensor installed on the camera; S2, constructing a spatial coordinate system offset correction parameter based on the obtained posture change of the camera, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system through the constructed spatial coordinate system offset correction parameter. The present invention solves the problem of lack of understanding of the camera state and the difficulty of calibration, and produces a technical effect of automatically monitoring the camera posture and automatically calibrating as needed.

Description

Technical Field

[0001] The present invention relates to the field of video monitoring technology, and in particular to a recalibration method of a camera perception system and a camera perception system. Background Art

[0002] Large cities will build national-level Internet of Vehicles pilot zones with high standards to achieve full coverage of the Internet of Vehicles. Based on this, new roads for vehicle-road collaboration will be built, intersections will be renovated, and new roadside communication units will be added. Among them, image perception equipment is widely used, and at least one camera is installed in each direction of the renovated intersections. In the context of large-scale construction of Internet of Vehicles systems, the number of cameras deployed on the roadside is gradually increasing.

[0003] Roadside cameras used for IoV target recognition need to be calibrated in advance. The main purpose of calibration is to calculate the conversion relationship between the camera pixel coordinate system and the GPS coordinate system. Only after the conversion relationship is established can the video recognition result correspond to the real GPS coordinates of the identified target. After the camera completes the initial calibration, its position and posture are not allowed to change. Otherwise, the calibrated correspondence will be destroyed, affecting the image recognition accuracy.

[0004] However, the cameras deployed on the roadside traffic lights are subject to long-term vibrations caused by vehicles and weather, and the camera posture will inevitably change. The tilt and rotation of the camera will cause the camera coordinate system to change. The change in the camera coordinate system will reduce the matching degree between the pixel coordinate system and the GPS coordinate system, ultimately leading to a decrease in image recognition rate and affecting the safety of the overall system.

[0005] In this case, the equipment operation and maintenance personnel usually go to the site for regular inspections to correct the tilt of the camera and recalibrate it. In this mode, it is difficult to detect problems in time and confirm the changes in the camera's posture, and there is a lack of remote monitoring and operation and maintenance methods.

[0006] There is an urgent need to use technical means to achieve real-time monitoring of the position and posture status of road-side perception cameras, and to recalibrate the camera's pixel coordinates and GPS coordinates through an automated feedback system. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a recalibration method for a camera perception system, which solves the problems of lack of real-time monitoring of the camera perception system and lack of efficient subsequent calibration in the prior art.

[0008] In a first aspect, according to an embodiment of the present invention, a recalibration method for a camera perception system is provided, which is used to calibrate a current virtual world coordinate system adopted by the camera perception system based on a posture change of the camera relative to an initial posture, wherein after the first calibration, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, including: S1, obtaining the posture change of the camera based on a tilt sensor installed on the camera; S2, constructing a spatial coordinate system offset correction parameter based on the obtained posture change of the camera, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed spatial coordinate system offset correction parameter.

[0009] According to an embodiment of the present invention, the posture change is the three-axis offset angle of the camera around the real world coordinate system.

[0010] According to an embodiment of the present invention, the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed space coordinate system offset correction parameter according to the following equation:

[0011]

[0012] Among them, DS1 is the spatial matrix representing the current virtual world coordinate system, R is the rotation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system, t is the translation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system, and DS0 is the spatial matrix representing the initial virtual world coordinate system.

[0013] According to an embodiment of the present invention, the rotation correction parameter is:

[0014] R=R x ×R y ×R z ,

[0015] Among them, R x , R y and R z for:

[0016]

[0017] Where i is x, y or z, θ i The offset angle of the camera around one of the three axes of the real-world coordinate system, x, y, or z.

[0018] According to an embodiment of the present invention, the translation correction parameter is:

[0019]

[0020] in,

[0021]

[0022]

[0023]

[0024] Among them, θ x ,θ y and θ z are the offset angles of the camera around one of the three axes of the real world coordinate system, x, y or z axis, respectively. L represents the straight-line distance between the center point of the camera base and the origin of the initial virtual world coordinate system. L is a constant and is obtained during the first calibration.

[0025] According to an embodiment of the present invention, the recalibration method also includes: calculating the initial volume of the camera's perception space based on the camera's initial posture; calculating the current volume of the camera's perception space based on the camera's posture change; calculating the volume ratio of the current volume of the perception space to the initial volume of the perception space, and comparing it with a preset ratio threshold; when the volume ratio is less than the ratio threshold, manually calibrating the camera.

[0026] According to an embodiment of the present invention, the recalibration method also includes: recording the offset angle of the camera around each real-world coordinate axis within a certain time period; calculating the variance of all offset angles of the camera around each real-world coordinate axis, and comparing the variance with a preset variance threshold; when the variance of all offset angles around any real-world coordinate axis is greater than the preset variance threshold, manually calibrating the camera.

[0027] According to an embodiment of the present invention, the variance is:

[0028]

[0029] Among them, λ is the variance of the camera's offset angle around any of the three axes of the real world coordinate system, τ is the offset angle sampling interval, N is the total number of sampling groups, and m is the random sampling range parameter, whose value range is θ K+2m ,θ K+m and θ K is the offset angle value of the corresponding sampling point.

[0030] According to an embodiment of the present invention, the time of the next recalibration is determined according to the following equation:

[0031]

[0032] Among them, T rIt represents the time interval from the last recalibration to the next recalibration, T is the time interval between the last two recalibrations, is the time interval correction constant, and the initial value of T is an artificially set engineering experience value.

[0033] On the other hand, according to an embodiment of the present invention, a camera perception system is provided for acquiring the real-world coordinates of objects in a video based on the video acquired by the camera and the current virtual world coordinate system, wherein after the camera perception system is installed and calibrated for the first time, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, comprising: a camera for acquiring video; a tilt sensor installed on the camera, for obtaining the posture change of the camera relative to the initial posture during calibration; and a calibration module, for obtaining the current virtual world coordinate system of the camera perception system based on the acquired posture change of the camera and the initial virtual world coordinate system.

[0034] The technical principle of the present invention is: using a sensor to obtain the posture change of the camera relative to the initial posture, calculating the spatial coordinate system offset correction parameter based on the posture change, and obtaining the current virtual world coordinate system of the camera perception system based on the initial virtual world coordinate system and the spatial coordinate system offset correction parameter.

[0035] Compared with the prior art, the present invention has the following beneficial effects: by adopting sensors to monitor the posture changes of the camera in real time, calculating the offset correction parameters of the spatial coordinate system based on the posture changes, and solving the current virtual world coordinate system of the camera perception system, it solves the problems of lack of real-time monitoring of the camera perception system and lack of efficient subsequent calibration technology, and produces the effect of being able to monitor the status of the perception system in real time and automatically calibrate when necessary, thereby ensuring the accuracy of the perception system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of a method for recalibrating a camera perception system according to an embodiment of the present invention.

[0037] Figure 2 It is a schematic diagram of valid area verification of perception field of view according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0039] As described in the background technology, the existing technology has the following problems: 1. There is a lack of camera posture monitoring means, and the camera changes cannot be detected in time; 2. The camera posture cannot be adjusted automatically after the change, and now it can only rely on the operation and maintenance personnel to go to the site to adjust the camera posture and recalibrate, which has high labor costs; 3. In the context of large-scale construction, it is urgent to choose a means to achieve a large number of equipment access and realize remote monitoring and operation. In order to solve the above problems, such as Figure 1 As shown, an embodiment of the present invention proposes a recalibration method for a camera perception system, which is used to calibrate the current virtual world coordinate system used by the camera perception system based on the posture change of the camera relative to the initial posture, wherein after the first calibration, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, including: S1, obtaining the posture change of the camera based on the tilt sensor installed on the camera; S2, constructing a spatial coordinate system offset correction parameter based on the obtained posture change of the camera, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system with the constructed spatial coordinate system offset correction parameter. By using a tilt sensor, the posture change of the camera can be monitored in real time, and the correction of the virtual world coordinate system used by the perception system can be calculated according to the posture change of the camera to reach an automatic calibration of the camera perception system, thereby realizing a camera perception system that integrates automatic monitoring and calibration without manual intervention.

[0040] According to an embodiment of the present invention, the posture change is the camera's three-axis offset angle around the real world coordinate system. Specifically, the offset angle is the camera's three-axis offset angle θ around the X / Y / Z axes. x ,θ y and θ z . The offset angle is obtained by the sensor. The sensor is installed between the road-end camera and the base, and fits tightly with the camera housing. Use products that are easy to install, simple to use, small in size, resistant to external electromagnetic interference, and strong in the ability to withstand vibration and shock. The tilt sensor mainly includes a three-axis high-precision inclinometer sensor that uses MEMS (Micro-Electro-Mechanical System) technology and digital output. It mainly converts changes in the static gravity field into inclination changes, and directly outputs the current three-axis angle changes in digital form. The measurement range is ±30° and the maximum accuracy is 0.001°.

[0041] In addition, the angle change data can be preprocessed to make the angle change curve smoother and more able to reflect the real changes of the camera. Affected by the complex environmental disturbances at the road end, the data obtained by the tilt sensor will contain a large number of interference signals. The data can be filtered and automatic compensation and filtering algorithms can be used to reduce the errors caused by environmental changes. After filtering and eliminating invalid and abnormal signals, the amount of data will also be reduced.

[0042] According to an embodiment of the present invention, the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed space coordinate system offset correction parameter according to the following equation:

[0043]

[0044] Among them, DS1 is the spatial matrix representing the current virtual world coordinate system, R is the rotation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system, t is the translation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system, and DS0 is the spatial matrix representing the initial virtual world coordinate system.

[0045] In detail, during the calibration process, after the camera is installed, the perception system establishes a virtual world coordinate system for the video it acquires based on the position and posture of the camera. In addition, the virtual world coordinates and GPS coordinates of the object in the video are obtained, and the virtual world coordinate system (called the initial virtual world coordinate system, represented by the space matrix DS0) is adjusted to correspond to the real world coordinate system, so that the real world coordinates of the object can be obtained according to the virtual world coordinates of the object in the video in the subsequent recognition.

[0046] After the camera is offset, the effect on the virtual world coordinate system can be calculated based on the camera offset, so that the corrected virtual world coordinate system (current coordinate system) corresponds to the real world coordinate system. Through this correction, the recognition accuracy of the perception system can be automatically improved without manual intervention, saving time and effort.

[0047] In addition, after a large number of statistical analyses of the current status of roadside cameras, it was found that the cameras installed on roadside poles for a long time mainly undergo rigid body changes. Rigid body transformation only changes the spatial position (translation) and orientation (rotation) of the data, but does not change its shape. It can be described by two variables: the orthogonal unit rotation matrix (rotation correction parameter) R, and the three-dimensional coordinate system translation vector (translation correction parameter) t. [R|t] is collectively referred to as the spatial coordinate system offset correction parameter. The spatial matrix representing the initial virtual world coordinate system is DS, and the spatial matrix representing the current virtual world coordinate system is DS1, then DS1 = [R|t]DS0.

[0048] In detail,

[0049]

[0050] Wherein, the rotation correction parameter is:

[0051] R=R x ×R y ×R z ,

[0052] Among them, R x , R y and R z for:

[0053]

[0054] Where i is x, y or z, θ i The offset angle of the camera around one of the three axes of the real-world coordinate system, x, y, or z.

[0055] The translation correction parameters are:

[0056]

[0057] in,

[0058]

[0059]

[0060]

[0061] Among them, θ x ,θ y and θ z are the offset angles of the camera around one of the three axes of the real world coordinate system, x, y or z axis, respectively. L represents the straight-line distance between the center point of the camera base and the origin of the initial virtual world coordinate system. L is a constant and is obtained during the first calibration.

[0062] In addition, when the camera is seriously tilted, a manual intervention mechanism is also set to prevent the automatic calibration method from failing to achieve the predetermined calibration effect. According to an embodiment of the present invention, the recalibration method further includes: calculating the initial volume of the camera's perception space based on the camera's initial posture; calculating the current volume of the camera's perception space based on the camera's posture change; calculating the volume ratio of the current volume of the perception space to the initial volume of the perception space, and comparing it with a preset ratio threshold; when the volume ratio is less than the ratio threshold, manually calibrating the camera.

[0063] like Figure 2As shown in the figure, the changes in the camera's perception space are shown. The blue area represents the camera's initial perception data space, the light green area represents the camera's perception data space after the offset, and the overlapping part can be understood as the effective part of the perception data space after the offset. The system relies on the offset parameter analysis obtained by the tilt sensor to obtain the specific volume size of the effective perception space.

[0064]

[0065] In the formula, V represents the effective field of view, V0 represents the initial perception data space of the camera, and V offset Indicates the effective perception data space of the camera after the offset. When the perception field efficiency is lower than the threshold set by the system, an offset warning will be triggered to remind the front-end maintenance personnel to perform manual calibration.

[0066] In addition, according to an embodiment of the present invention, the recalibration method also monitors the change in the offset angle to perform manual calibration when necessary. In detail, the recalibration method also includes: recording the offset angle of the camera around each real-world coordinate axis within a certain period of time; calculating the variance of all offset angles of the camera around each real-world coordinate axis, and comparing the variance with a preset variance threshold; when the variance of all offset angles around any real-world coordinate axis is greater than the preset variance threshold, performing manual maintenance on the camera. The variance is:

[0067]

[0068] Among them, λ is the variance of the camera's offset angle around any of the three axes of the real world coordinate system, τ is the offset angle sampling interval, N is the total number of sampling groups, and m is the random sampling range parameter, whose value range is θ K+2m ,θ K+m and θ K is the offset angle value of the corresponding sampling point.

[0069] When the λ(*) value exceeds the threshold, it indicates that the camera tilt changes too quickly. For equipment systems that change too quickly, that is, the tilt trend is increasing, special attention should be paid to it and a high-risk alert should be triggered. At the same time, the backend operation and maintenance personnel should be reminded to pay attention to the equipment situation in the area. The camera in this area may be more likely to tilt due to the surrounding environment and structural problems, and reinforcement measures should be taken during the next on-site maintenance.

[0070] In addition, the frequency or time interval of automatic calibration can also be customized. According to an embodiment of the present invention, the time of the next recalibration is determined according to the following equation:

[0071]

[0072] Among them, T r It represents the time interval from the last recalibration to the next recalibration, T is the time interval between the last two recalibrations, is the time interval correction constant, and the initial value of T is an artificially set engineering experience value.

[0073] According to another embodiment of the present invention, a camera perception system is provided, which is used to obtain the real-world coordinates of objects in the video based on the video acquired by the camera and the current virtual world coordinate system, wherein after the camera perception system is installed and calibrated for the first time, the coordinates of the same objects in the initial virtual world coordinate system and the real world coordinate system are the same, including: a camera, used to acquire video; a tilt sensor, installed on the camera, used to obtain the posture change of the camera relative to the initial posture during calibration; a calibration module, used to obtain the current virtual world coordinate system of the camera perception system based on the obtained posture change of the camera and the initial virtual world coordinate system.

[0074] The present invention adopts sensors to monitor the posture changes of the camera in real time, calculates the offset correction parameters of the spatial coordinate system based on the posture changes, and solves the current virtual world coordinate system of the camera perception system. It solves the problems of lack of real-time monitoring of the camera perception system and lack of efficient subsequent calibration technology, and produces the effect of being able to monitor the status of the perception system in real time and automatically calibrate when necessary, ensuring the accuracy of the perception system while saving time and effort.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for recalibrating a camera perception system, for calibrating a current virtual world coordinate system used by the camera perception system based on a change in the camera's posture relative to an initial posture, wherein: After the first calibration, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, which is characterized by including: S1, obtaining a posture change of the camera based on a tilt sensor installed on the camera; S2. constructing a spatial coordinate system offset correction parameter based on the obtained camera posture change, so that the coordinates of the same object in the current virtual world coordinate system and the real world coordinate system are the same, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed spatial coordinate system offset correction parameter; The posture change is the three-axis offset angle of the camera around the real world coordinate system; The current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed space coordinate system offset correction parameter according to the following equation: ， in, is the space matrix representing the current virtual world coordinate system, is the rotation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system. is the translation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system. is the space matrix representing the initial virtual world coordinate system.

2. A method for recalibrating a camera perception system as claimed in claim 1, characterized in that: The rotation correction parameters are: ， in, , and for: ， in, for , or , The camera rotates around one of the three axes of the real world coordinate system , or The offset angle.

3. A method for recalibrating a camera perception system as claimed in claim 1, characterized in that: The translation correction parameters are: ， in, ， ， ， in, , and They are respectively one of the three axes of the camera around the real world coordinate system , or The offset angle of the axis, represents the straight-line distance between the center point of the camera base and the origin of the initial virtual world coordinate system, and is a constant and is obtained during the first calibration.

4. A method for recalibrating a camera perception system as claimed in claim 1, characterized in that: The recalibration method further comprises: Calculate the initial volume of the camera's perception space based on the camera's initial posture; Calculate the current volume of the camera's perception space based on the camera's posture change; Calculating a volume ratio of the current volume of the perception space to the initial volume of the perception space, and comparing the ratio with a preset ratio threshold; When the volume ratio is less than a ratio threshold, the camera is manually calibrated.

5. A method for recalibrating a camera perception system as claimed in claim 1, characterized in that: The recalibration method further comprises: Record the camera's offset angle around each real-world coordinate axis within a certain period of time; Calculating the variance of all offset angles of the camera around each real-world coordinate axis, and comparing the variance with a preset variance threshold; When the variance of all offset angles around any real-world coordinate axis is greater than a preset variance threshold, the camera is manually calibrated.

6. A method for recalibrating a camera perception system as claimed in claim 5, characterized in that: The variance is: in, is the variance of the offset angle, which is the offset angle of the camera around any of the three axes of the real world coordinate system. is the offset angle sampling interval, is the total number of sampling groups, is the random sampling range parameter, and its value range is , , and is the offset angle value of the corresponding sampling point.

7. A method for recalibrating a camera perception system as claimed in claim 6, characterized in that: The time for the next recalibration is determined by the following equation: ， in, Indicates the time interval from the last recalibration to the next recalibration. is the time interval between the last two recalibrations, is the time interval correction constant, and The initial value is an artificially set engineering experience value.

8. A camera perception system for obtaining real-world coordinates of an object in a video based on a video acquired by a camera and a current virtual world coordinate system, wherein: After the camera perception system is first installed and calibrated, the coordinates of the same object in the initial virtual world coordinate system and the real world coordinate system are the same, and the characteristics include: A camera, used to acquire video; A tilt sensor, mounted on the camera, for obtaining a change in the camera's posture relative to an initial posture during calibration; A calibration module, used to obtain a current virtual world coordinate system of the camera perception system based on the obtained camera posture change and the initial virtual world coordinate system, wherein the current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed space coordinate system offset correction parameter; The posture change is the three-axis offset angle of the camera around the real world coordinate system; The current virtual world coordinate system is obtained by correcting the initial virtual world coordinate system by the constructed space coordinate system offset correction parameter according to the following equation: ， in, is the space matrix representing the current virtual world coordinate system, is the rotation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system. is the translation correction parameter calculated based on the three-axis offset angle of the camera around the real world coordinate system. is the space matrix representing the initial virtual world coordinate system.

Images