Camera extrinsic parameter calibration method, device, equipment, medium and program product

Abstract

The present application provides a camera external parameter calibration method, apparatus, equipment, medium and program product. When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, and the calibration prompt message is used to instruct the user to perform preset visual interaction actions on different first calibration external parameters multiple times, and the first calibration external parameter is located outside the current image acquisition range of the camera to be calibrated; a calibration image of the user performing the preset visual interaction action is obtained through the camera to be calibrated; an image analysis model is used to determine the relative position information between the camera to be calibrated, the user and each first calibration external parameter based on the calibration image; and the external parameter conversion matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship. This solves the technical problem in the prior art that the camera external parameter self-calibration is easily affected by environmental changes, resulting in self-calibration failure. It achieves the technical effect of improving the anti-interference performance of camera calibration and the accuracy of subsequent image analysis.

Description

Technical Field

[0001] The present application relates to the field of monitoring equipment, and in particular to a camera external parameter calibration method, device, equipment, medium and program product. Background Art

[0002] With the development of science and technology and the improvement of people's living standards, intelligence has become the development trend of many products. An important part of realizing intelligence is to detect and monitor user behavior in real time. This is inseparable from the participation of cameras.

[0003] Analyzing images captured by a camera involves transforming the camera's coordinate system to the preset coordinate system of the environment in which it resides. This transformation is accomplished using an extrinsic transformation matrix. Furthermore, if the camera's installation position changes, the extrinsic transformation matrix must be recalibrated to ensure the accuracy of the image analysis results.

[0004] However, existing calibration methods generally utilize fixed, unchanging image features within the camera's captured image. However, these methods often fail due to the loss of fixed image features, such as when a standard reference object is obscured or removed from the image. This results in self-calibration failure. This is because existing camera extrinsic self-calibration is susceptible to environmental changes, leading to self-calibration failures. Summary of the Invention

[0005] The present application provides a camera extrinsic parameter calibration method, device, equipment, medium and program product to solve the technical problem in the prior art that camera extrinsic parameter self-calibration is easily affected by environmental changes, resulting in self-calibration failure.

[0006] In a first aspect, the present application provides a camera extrinsic parameter calibration method, comprising:

[0007] When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, where the calibration prompt message is used to instruct the user to perform a preset visual interaction action multiple times on different first calibration extrinsic parameters, where the first calibration extrinsic parameter is outside the current image acquisition range of the camera to be calibrated;

[0008] Acquire a calibration image when the user performs a preset visual interaction action through the camera to be calibrated;

[0009] Using the image analysis model, the relative position information between the camera to be calibrated, the user, and each first calibration external reference is determined based on the calibration image;

[0010] The extrinsic parameter transformation matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship.

[0011] In one possible design, an image analysis model is used to determine the relative position information between the camera to be calibrated, the user, and each first calibration extrinsic parameter based on the calibration image, including:

[0012] Using an image analysis model, based on the calibration image, determine first relative position information between the user and the camera to be calibrated, and second relative position information between the user and a first calibration extrinsic parameter, the relative position information including: first relative position information and second relative position information;

[0013] Correspondingly, the extrinsic parameter conversion matrix after recalibration of the camera to be calibrated is determined based on the relative position relationship, including:

[0014] Determine the current position coordinates of the camera to be calibrated in the preset coordinate system according to the position coordinates of each first calibration extrinsic parameter in the preset coordinate system, the first relative position information, and each second relative position information;

[0015] According to the current position coordinates and the latest historical position coordinates of the camera to be calibrated, the external parameter transformation matrix is ​​corrected and transformed to recalibrate the external parameter transformation matrix.

[0016] In a possible design, the position of each first calibration extrinsic parameter in the preset coordinate system is constant, and the position of the camera to be calibrated in the preset coordinate system is variable.

[0017] In one possible design, the first relative position relationship includes: a first relative direction and / or a first relative distance between the user and the camera to be calibrated, and the second relative position relationship includes: a second relative direction and / or a second relative distance between the user and the first calibration external parameter.

[0018] In one possible design, the extrinsic parameter transformation matrix is ​​modified and transformed based on the current position coordinates and the latest historical position coordinates of the camera to be calibrated to recalibrate the extrinsic parameter transformation matrix, including:

[0019] Calculate the coordinate offset vector of the current position coordinates and the historical position coordinates;

[0020] The extrinsic transformation matrix is ​​transformed according to the coordinate offset vector to recalibrate the extrinsic transformation matrix.

[0021] In a possible design, the preset visual interaction action includes: looking at a first calibration external parameter.

[0022] Optionally, looking at the first calibration external reference includes: the user facing forward and turning eyes toward the first calibration external reference, or the user facing the first calibration external reference and looking directly at the first calibration external reference with eyes.

[0023] In a possible design, each first calibration extrinsic parameter is provided with at least one of: a display, a light emitter, a reflector, and a light diffuser.

[0024] In a possible design, detecting that the camera to be calibrated fails in self-calibration includes:

[0025] Obtain the user's current gaze area and interested gaze area. The current gaze area is determined based on the real-time image data collected by the camera to be calibrated. The first position coordinates of the camera to be calibrated in the preset coordinate system are variable. The interested gaze area is the area where the user's gaze probability is greater than a preset threshold when performing the preset task multiple times.

[0026] Determine whether the deviation between the current gaze area and the interested gaze area is greater than a preset deviation threshold;

[0027] If so, it is determined that the extrinsic parameter conversion matrix of the camera to be calibrated needs to be calibrated and corrected.

[0028] In a possible design, before obtaining the user's current gaze area, the following is also included:

[0029] When the camera to be calibrated is powered on, a self-calibration mode of the camera to be calibrated is started, so that the camera to be calibrated performs self-calibration according to preset features in the collected real-time image;

[0030] If the self-calibration fails after N consecutive repetitions, the extrinsic parameter conversion matrix of the camera to be calibrated is set to the historical extrinsic parameter conversion matrix obtained when the self-calibration was successful last time.

[0031] In one possible design, the camera to be calibrated is a surveillance camera of an anti-fatigue monitoring system.

[0032] In a second aspect, the present application provides a camera extrinsic parameter calibration device, comprising:

[0033] The acquisition module is used to obtain the self-calibration status of the camera to be calibrated;

[0034] Processing module for:

[0035] When it is detected according to the self-calibration situation that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, where the calibration prompt message is used to instruct the user to perform a preset visual interaction action multiple times on different first calibration extrinsic parameters, where the first calibration extrinsic parameter is outside the current image acquisition range of the camera to be calibrated;

[0036] Acquire a calibration image when the user performs a preset visual interaction action through the camera to be calibrated;

[0037] Using the image analysis model, the relative position information between the camera to be calibrated, the user, and each first calibration external reference is determined based on the calibration image;

[0038] The extrinsic parameter transformation matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship.

[0039] In one possible design, the processing module is configured to:

[0040] Using an image analysis model, based on the calibration image, determine first relative position information between the user and the camera to be calibrated, and second relative position information between the user and a first calibration extrinsic parameter, the relative position information including: first relative position information and second relative position information;

[0041] Determine the current position coordinates of the camera to be calibrated in the preset coordinate system according to the position coordinates of each first calibration extrinsic parameter in the preset coordinate system, the first relative position information, and each second relative position information;

[0042] According to the current position coordinates and the latest historical position coordinates of the camera to be calibrated, the external parameter transformation matrix is ​​corrected and transformed to recalibrate the external parameter transformation matrix.

[0043] In a possible design, the position of each first calibration extrinsic parameter in the preset coordinate system is constant, and the position of the camera to be calibrated in the preset coordinate system is variable.

[0044] In one possible design, the first relative position relationship includes: a first relative direction and / or a first relative distance between the user and the camera to be calibrated, and the second relative position relationship includes: a second relative direction and / or a second relative distance between the user and the first calibration external parameter.

[0045] In one possible design, the processing module is configured to:

[0046] Calculate the coordinate offset vector of the current position coordinates and the historical position coordinates;

[0047] The extrinsic transformation matrix is ​​transformed according to the coordinate offset vector to recalibrate the extrinsic transformation matrix.

[0048] In a possible design, the preset visual interaction action includes: looking at a first calibration external parameter.

[0049] Optionally, looking at the first calibration external reference includes: the user facing forward and turning eyes toward the first calibration external reference, or the user facing the first calibration external reference and looking directly at the first calibration external reference with eyes.

[0050] In a possible design, each first calibration extrinsic parameter is provided with at least one of: a display, a light emitter, a reflector, and a light diffuser.

[0051] In one possible design, the acquisition module is used to obtain the user's current gaze area and the interested gaze area. The current gaze area is determined based on real-time image data collected by the camera to be calibrated. The first position coordinates of the camera to be calibrated in a preset coordinate system are variable. The interested gaze area is an area where the probability of the user's gaze being fixed is greater than a preset threshold when the user performs the preset task multiple times.

[0052] The processing module is also used to:

[0053] Determine whether the deviation between the current gaze area and the interested gaze area is greater than a preset deviation threshold;

[0054] If so, it is determined that the extrinsic parameter conversion matrix of the camera to be calibrated needs to be calibrated and corrected.

[0055] In a possible design, the processing module is further configured to:

[0056] When the camera to be calibrated is powered on, a self-calibration mode of the camera to be calibrated is started, so that the camera to be calibrated performs self-calibration according to preset features in the collected real-time image;

[0057] If the self-calibration fails after N consecutive repetitions, the extrinsic parameter conversion matrix of the camera to be calibrated is set to the historical extrinsic parameter conversion matrix obtained when the self-calibration was successful last time.

[0058] In one possible design, the camera to be calibrated is a surveillance camera of an anti-fatigue monitoring system.

[0059] In a third aspect, the present application provides an electronic device, comprising:

[0060] a memory for storing program instructions;

[0061] The processor is used to call and execute the program instructions in the memory to perform any possible camera extrinsic parameter calibration method provided by the first aspect.

[0062] In a fourth aspect, the present application provides an anti-fatigue monitoring system, comprising: at least one monitoring camera and any possible electronic device provided in the third aspect; wherein the electronic device is used to calibrate the extrinsic parameter conversion matrix of the monitoring camera to implement any possible camera extrinsic parameter calibration method provided in the first aspect.

[0063] In a fifth aspect, the present application provides a vehicle comprising any possible anti-fatigue monitoring system provided in the fourth aspect.

[0064] In a sixth aspect, the present application provides a storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to execute any possible camera extrinsic parameter calibration method provided in the first aspect.

[0065] In the seventh aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any possible camera extrinsic parameter calibration system method provided in the first aspect.

[0066] The present application provides a camera external parameter calibration method, apparatus, equipment, medium and program product. When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, and the calibration prompt message is used to instruct the user to perform preset visual interaction actions on different first calibration external parameters multiple times, and the first calibration external parameter is located outside the current image acquisition range of the camera to be calibrated; a calibration image of the user performing the preset visual interaction action is obtained through the camera to be calibrated; an image analysis model is used to determine the relative position information between the camera to be calibrated, the user and each first calibration external parameter based on the calibration image; and the external parameter conversion matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship. This solves the technical problem in the prior art that the camera external parameter self-calibration is easily affected by environmental changes, resulting in self-calibration failure. It achieves the technical effect of improving the anti-interference performance of camera calibration and the accuracy of subsequent image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of the structure of a wearable device provided in this application;

[0069] Figure 2 A flowchart of a camera extrinsic parameter calibration method provided in an embodiment of the present application;

[0070] Figure 3 A flowchart of another camera extrinsic parameter calibration method provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a camera extrinsic parameter calibration device provided in an embodiment of the present application;

[0072] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application.

[0073] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0074] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts, including but not limited to combinations of multiple embodiments, are within the scope of protection of this application.

[0075] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] The fundamental logic of modern artificial intelligence is to use cameras to collect real-time data on user behavior, expressions, and postures, thereby analyzing their behavioral intentions and enabling AI-powered control of the products they use. This makes image recognition and analysis a crucial technical foundation of artificial intelligence. During image recognition, the camera's coordinate system and the world coordinate system of its environment require a reference object to calibrate the conversion medium, known as the extrinsic conversion matrix. This allows the image recognition results to be converted to the corresponding real-world results in the world coordinate system.

[0077] In actual use, we found that cameras are sometimes installed on movable modules or devices, which requires recalibrating the external parameter conversion matrix when the camera starts working. For example, if the monitoring camera is installed on the steering column to monitor whether the driver is driving fatigued, if the driver adjusts the position of the steering wheel, such as extending or swinging it downward, in order to correctly identify the driver's true line of sight, the external parameter conversion matrix needs to be recalibrated. However, when recalibrating the camera, the existing technology generally relies on a fixed reference object in the camera acquisition image, such as an interior decoration part at the driving position. However, as user personalized needs become increasingly high, users may make personalized modifications during product use, such as changing the vehicle interior or adding other decorations, which obscures the reference object. This makes it impossible for the camera to find the corresponding preset reference feature in the image during self-calibration, resulting in self-calibration failure, which in turn affects the normal use of other functions of artificial intelligence.

[0078] In order to solve the above technical problems, the invention concept of this application is:

[0079] Outside the monitoring screen area of ​​the camera, at a certain distance away from the user, a calibration external parameter that is not easily affected by the user's personalized behavior is set. This calibration external parameter is completely separated from the camera, and its position in the environment, or in other words, its position in the world coordinate system (hereinafter also referred to as the preset coordinate system) is unchanged. After the camera fails to perform self-calibration, a prompt message is output to allow the user to visually interact with one or more calibration external parameters. In this way, by analyzing the user image captured by the camera, the relative position relationship between the calibration external parameter, the user and the camera is obtained, and then the current coordinates of the camera in the preset coordinate system are reversed based on the known coordinates of the calibration external parameter. By comparing the coordinates of the camera at the time of the last successful calibration with the current coordinates, the external parameter conversion matrix can be modified to achieve recalibration of the external parameter conversion matrix.

[0080] The following is a detailed introduction on how this application realizes the transmission of information through touch.

[0081] Figure 1 This is a schematic diagram of an application scenario of a camera extrinsic calibration method provided in this application. Figure 1As shown, camera 100 is a surveillance camera for the vehicle's anti-fatigue monitoring system. Mounted on the steering column or the steering column's trim, it captures a spatial image of the driver's seat and surrounding areas. Typically, camera 100 performs self-calibration upon power-up, using the interior features of the driver's seat and surrounding areas as a calibration reference. If these interior features are obscured or modified by the vehicle owner, the surveillance camera's self-calibration fails. In this case, the driver can be prompted to focus on a first calibration extrinsic reference 200 located outside the vehicle's monitoring range. By analyzing the user's gaze direction, the relative positional relationship between camera 100, first calibration extrinsic reference 200, and the driver can be determined. The coordinates of camera 100 can then be inferred from the coordinates of first calibration extrinsic reference 200 in a preset coordinate system. By transforming the camera 100's coordinates, the changes in the extrinsic parameter transformation matrix can be determined, allowing the extrinsic parameter transformation matrix to be recalibrated.

[0082] It should be noted that Figure 1 The gray dotted box in represents a display for displaying the first calibration external parameter 200, which can be understood as a central control display screen or a display area of ​​a head-up display.

[0083] Figure 2 This is a flow chart of a camera extrinsic calibration method provided in an embodiment of the present application. Figure 2 As shown, the specific steps of the camera extrinsic parameter calibration method include:

[0084] S201: When it is detected that the camera to be calibrated fails in self-calibration, a calibration prompt message is output.

[0085] In this step, the calibration prompt information is used to instruct the user to perform preset visual interaction actions on different first calibration extrinsic parameters multiple times, and the first calibration extrinsic parameter is located outside the current image acquisition range of the camera to be calibrated.

[0086] Specifically, the camera to be calibrated can automatically start the self-calibration mode every time it is powered on to try to recalibrate the external parameter conversion matrix, or start the self-calibration mode after receiving a self-calibration instruction from the user or the controller. The self-calibration mode generally relies on the image features in the monitoring image of the camera to be calibrated itself, such as the outline of a preset reference object in the image for self-calibration. When the image features are artificially blocked or modified, the self-calibration mode will lose its function, that is, the self-calibration fails. At this time, the corresponding prompt information is output through voice or displayed on the display screen, such as "The self-calibration of the surveillance camera failed, please perform manual auxiliary calibration according to the prompt..." Then, the user needs to visually interact with multiple first calibration external parameters according to the prompt information, including: looking at the first calibration external parameters.

[0087] Optionally, looking at the first calibration external reference includes: the user facing forward and turning eyes toward the first calibration external reference, or the user facing the first calibration external reference and looking directly at the first calibration external reference with eyes.

[0088] It should be noted that in one possible design, each first calibration extrinsic parameter is provided with at least one of: a display, a light emitter, a reflector, and a light diffuser. For example, the first calibration extrinsic parameter may be an indicator light located at a certain position on a central control display screen within a vehicle, or a geometric figure displayed on the display screen, such as a dot in the upper right corner of the central control display screen. Another example may be one or more small squares projected by a projection device in a head-up display area within a vehicle or on an aircraft.

[0089] It should also be noted that there is no positional coupling relationship between the first calibration external parameter and the camera to be calibrated, or in other words, the first calibration external parameter and the camera to be calibrated are not installed on the same module or device, and the coordinate position of the first calibration external parameter in the preset coordinate system (such as the vehicle coordinate system to which the anti-fatigue monitoring system belongs) is fixed.

[0090] S202: Obtain, through the camera to be calibrated, a calibration image of the user performing a preset visual interaction action.

[0091] In this step, the calibration image includes a monitoring screen when the user performs a preset visual interaction action, and each first calibration extrinsic parameter corresponds to at least one calibration image.

[0092] S203: Using the image analysis model, determine the relative position information among the camera to be calibrated, the user, and each first calibration external parameter according to the calibration image.

[0093] In this step, the image analysis model is used to determine the first relative position information between the user and the camera to be calibrated, as well as the second relative position information between the user and a first calibration external parameter based on the calibration image. The relative position information includes: first relative position information and second relative position information.

[0094] It should be noted that the number of relative position information items corresponds one-to-one to the number of first calibration extrinsic parameters. That is, in this step, by analyzing the calibration image corresponding to each first calibration extrinsic parameter in S202, the second relative position information between each first calibration extrinsic parameter and the user, as well as the first relative position information between the user and the camera to be calibrated, is obtained. Thus, through the user as an intermediary, the positional relationship between the camera to be calibrated and the first calibration extrinsic parameter is established. Thus, the coordinates of the camera to be calibrated can be inferred from the known coordinates of the first calibration extrinsic parameter in the preset coordinate system.

[0095] In order to improve the accuracy of the relative position information, the coordinates of the camera to be calibrated can be obtained multiple times by setting multiple first calibration external parameters. Finally, the coordinate values ​​are screened, and after removing the values ​​with large deviations, the average value is taken as the coordinates of the camera to be calibrated.

[0096] On the other hand, the number of first calibration extrinsic parameters is also related to the number of dimensions of the coordinates of the camera to be calibrated. If the preset coordinate system is two-dimensional, then at least two first calibration extrinsic parameters are required. If the preset coordinate system is three-dimensional, then at least three first calibration extrinsic parameters are required to determine the coordinates of the camera to be calibrated.

[0097] S204: Determine the extrinsic parameter conversion matrix after recalibration of the camera to be calibrated according to the relative position relationship.

[0098] In this step, the coordinates of the camera to be calibrated in the preset coordinate system and their relative positional relationship in the preset coordinate system are used to reversely deduce the coordinates of the camera to be calibrated in the preset coordinate system through geometric inversion. These coordinates are then compared with the coordinates of the camera to be calibrated during its last successful calibration to obtain the position offset vector of the camera to be calibrated. This position offset vector is then used to modify the extrinsic transformation matrix of the camera to be calibrated during its last successful calibration to obtain the recalibrated extrinsic transformation matrix, thus achieving recalibration of the camera to be calibrated.

[0099] This embodiment provides a camera external parameter calibration system method. When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output. The calibration prompt message is used to instruct the user to perform preset visual interaction actions on different first calibration external parameters multiple times. The first calibration external parameter is located outside the current image acquisition range of the camera to be calibrated; a calibration image of the user performing the preset visual interaction action is obtained through the camera to be calibrated; an image analysis model is used to determine the relative position information between the camera to be calibrated, the user, and each first calibration external parameter based on the calibration image; and the external parameter conversion matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship. This solves the technical problem in the prior art that the camera external parameter self-calibration is easily affected by environmental changes, resulting in self-calibration failure. The technical effect of improving the anti-interference performance of camera calibration and the accuracy of subsequent image analysis is achieved.

[0100] Figure 3 This is a flow chart of another camera extrinsic parameter calibration method provided in an embodiment of the present application. Figure 3 As shown, the specific steps of the camera extrinsic parameter calibration method include:

[0101] S301 . When the camera to be calibrated is powered on, start the self-calibration mode of the camera to be calibrated.

[0102] In this step, the self-calibration mode is started to enable the camera to be calibrated to perform self-calibration according to preset features in the collected real-time image.

[0103] In this embodiment, the camera to be calibrated is a monitoring camera for an anti-fatigue monitoring system on a vehicle. Figure 1 As shown, camera 100 is the monitoring camera of the anti-fatigue monitoring system. It is installed on the steering column. Because the steering column can be adjusted in two directions, the position of camera 100 also changes. This requires recalibrating the extrinsic parameter conversion matrix when analyzing the image data collected by camera 100 to ensure the accuracy of the analysis results. Therefore, each time the camera to be calibrated is powered on, it is set to automatically start the self-calibration mode.

[0104] In self-calibration mode, by identifying preset features in the real-time image, such as the vehicle's interior features, which have been pre-calibrated by the manufacturer in the vehicle coordinate system, the current coordinates of the camera to be calibrated can be obtained.

[0105] S302: If the self-calibration fails after being repeated N times, the extrinsic parameter conversion matrix of the camera to be calibrated is set to the historical extrinsic parameter conversion matrix obtained when the self-calibration was successful last time.

[0106] In this step, the camera to be calibrated is self-calibrated through the self-calibration mode. However, if the calibration fails for N consecutive times, such as 5 or 10 times, it proves that the camera to be calibrated cannot find the corresponding preset feature within the monitoring range, that is, the preset feature may be removed or modified, or blocked, or the current position of the camera to be calibrated may have deviated significantly from the previous preset installation position.

[0107] In this embodiment, after self-calibration fails, the external parameter conversion matrix during the last successful self-calibration, i.e., the historical external parameter conversion matrix, is directly used to ensure that the fatigue monitoring function can still be activated and output the driver's gaze direction, head posture, facial attributes and other information.

[0108] S303: Acquire the user's current gaze area and interested gaze area.

[0109] In this step, the current gaze area is determined based on the real-time image data collected by the camera to be calibrated. The first position coordinates of the camera to be calibrated in the preset coordinate system are variable. The interest gaze area is the area where the probability of the user's gaze being observed when performing the preset task multiple times is greater than the preset threshold.

[0110] It should be noted that the area of ​​interest is a physical area determined based on the probability density distribution of the user's gaze area during driving, obtained from a large amount of statistical data. After the user or driver starts the vehicle and the calibration camera is powered on, facial recognition technology is used to determine the driver's identity information. Based on this identity information, the user's corresponding area of ​​interest is retrieved from the backend or cloud database.

[0111] S304: Determine whether the deviation between the current gaze area and the interested gaze area is greater than a preset deviation threshold.

[0112] In this step, if yes, it is determined that the extrinsic parameter conversion matrix of the camera to be calibrated needs to be calibrated and corrected, and the process goes to the next step S305; if no, there is no need to recalibrate the camera.

[0113] Specifically, the system calculates the deviation between the current gaze area and the area of ​​interest during a period of driving. When the deviation exceeds a preset deviation threshold, the anti-fatigue function (including: line of sight screen lighting, driver attention monitoring, etc.) is suppressed to avoid misjudgment that affects the driver's normal driving.

[0114] S305: Output calibration prompt information.

[0115] In this step, the calibration prompt information is used to instruct the user to perform preset visual interaction actions on different first calibration extrinsic parameters multiple times, and the first calibration extrinsic parameter is located outside the current image acquisition range of the camera to be calibrated.

[0116] Specifically, under the premise of ensuring safe driving, such as when the vehicle speed is lower than the preset safety speed, the pre-designed calibration point will be lit in the display area of ​​the central control screen or head-up display, and a prompt message will be displayed or played on the instrument panel to remind the driver to look at the currently lit calibration point.

[0117] S306: Acquire, through the camera to be calibrated, a calibration image when the user performs a preset visual interaction action.

[0118] S307 : Using the image analysis model, determine first relative position information between the user and the camera to be calibrated, and second relative position information between the user and a first calibration extrinsic parameter according to the calibration image.

[0119] S308 : Determine the current position coordinates of the camera to be calibrated in the preset coordinate system according to the position coordinates of each first calibration extrinsic parameter in the preset coordinate system, the first relative position information, and each second relative position information.

[0120] S309 , performing a correction transformation on the extrinsic parameter conversion matrix according to the current position coordinates and the latest historical position coordinates of the camera to be calibrated, so as to recalibrate the extrinsic parameter conversion matrix.

[0121] In this embodiment, in steps S306-S309, as the driver gazes at different calibration points, the first calibration extrinsic parameter, the camera to be calibrated calculates the relative position of the face (or eyes) and the camera to be calibrated, as well as the driver's current gaze direction. Combined with the known calibration point positions, the relative positional relationship between the camera to be calibrated and the first calibration extrinsic parameter is calculated, thereby obtaining the camera transformation matrix, or extrinsic parameter transformation matrix. At this point, calibration is complete, the extrinsic parameter transformation matrix is ​​updated, and the current user's gaze area of ​​interest is continuously monitored and counted.

[0122] This embodiment provides a camera external parameter calibration system method. When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output. The calibration prompt message is used to instruct the user to perform preset visual interaction actions on different first calibration external parameters multiple times. The first calibration external parameter is located outside the current image acquisition range of the camera to be calibrated; a calibration image of the user performing the preset visual interaction action is obtained through the camera to be calibrated; an image analysis model is used to determine the relative position information between the camera to be calibrated, the user, and each first calibration external parameter based on the calibration image; and the external parameter conversion matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship. This solves the technical problem in the prior art that the camera external parameter self-calibration is easily affected by environmental changes, resulting in self-calibration failure. The technical effect of improving the anti-interference performance of camera calibration and the accuracy of subsequent image analysis is achieved.

[0123] Figure 4 This is a schematic diagram of the structure of a camera extrinsic parameter calibration device provided in an embodiment of the present application. The camera extrinsic parameter calibration device 400 can be implemented by software, hardware, or a combination of both.

[0124] like Figure 4 As shown, the camera extrinsic parameter calibration device 400 includes:

[0125] An acquisition module 401 is used to obtain the self-calibration status of the camera to be calibrated;

[0126] The processing module 402 is configured to:

[0127] When it is detected according to the self-calibration situation that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, where the calibration prompt message is used to instruct the user to perform a preset visual interaction action multiple times on different first calibration extrinsic parameters, where the first calibration extrinsic parameter is outside the current image acquisition range of the camera to be calibrated;

[0128] Acquire a calibration image when the user performs a preset visual interaction action through the camera to be calibrated;

[0129] Using the image analysis model, the relative position information between the camera to be calibrated, the user, and each first calibration external reference is determined based on the calibration image;

[0130] The extrinsic parameter transformation matrix of the camera to be calibrated after recalibration is determined based on the relative position relationship.

[0131] In one possible design, the processing module 402 is configured to:

[0132] Using an image analysis model, based on the calibration image, determine first relative position information between the user and the camera to be calibrated, and second relative position information between the user and a first calibration extrinsic parameter, the relative position information including: first relative position information and second relative position information;

[0133] Determine the current position coordinates of the camera to be calibrated in the preset coordinate system according to the position coordinates of each first calibration extrinsic parameter in the preset coordinate system, the first relative position information, and each second relative position information;

[0134] According to the current position coordinates and the latest historical position coordinates of the camera to be calibrated, the external parameter transformation matrix is ​​corrected and transformed to recalibrate the external parameter transformation matrix.

[0135] In a possible design, the position of each first calibration extrinsic parameter in the preset coordinate system is constant, and the position of the camera to be calibrated in the preset coordinate system is variable.

[0136] In one possible design, the first relative position relationship includes: a first relative direction and / or a first relative distance between the user and the camera to be calibrated, and the second relative position relationship includes: a second relative direction and / or a second relative distance between the user and the first calibration external parameter.

[0137] In one possible design, the processing module 402 is configured to:

[0138] Calculate the coordinate offset vector of the current position coordinates and the historical position coordinates;

[0139] The extrinsic transformation matrix is ​​transformed according to the coordinate offset vector to recalibrate the extrinsic transformation matrix.

[0140] In a possible design, the preset visual interaction action includes: looking at a first calibration external parameter.

[0141] Optionally, looking at the first calibration external reference includes: the user facing forward and turning eyes toward the first calibration external reference, or the user facing the first calibration external reference and looking directly at the first calibration external reference with eyes.

[0142] In a possible design, each first calibration extrinsic parameter is provided with at least one of: a display, a light emitter, a reflector, and a light diffuser.

[0143] In one possible design, acquisition module 401 is configured to acquire a user's current gaze area and an interested gaze area. The current gaze area is determined based on real-time image data captured by the camera to be calibrated. The first position coordinates of the camera to be calibrated in a preset coordinate system are variable. The interested gaze area is an area where the probability of the user's gaze being observed when performing a preset task multiple times is greater than a preset threshold.

[0144] The processing module 402 is further configured to:

[0145] Determine whether the deviation between the current gaze area and the interested gaze area is greater than a preset deviation threshold;

[0146] If so, it is determined that the extrinsic parameter conversion matrix of the camera to be calibrated needs to be calibrated and corrected.

[0147] In one possible design, the processing module 402 is further configured to:

[0148] When the camera to be calibrated is powered on, a self-calibration mode of the camera to be calibrated is started, so that the camera to be calibrated performs self-calibration according to preset features in the collected real-time image;

[0149] If the self-calibration fails after N consecutive repetitions, the extrinsic parameter conversion matrix of the camera to be calibrated is set to the historical extrinsic parameter conversion matrix obtained when the self-calibration was successful last time.

[0150] In one possible design, the camera to be calibrated is a surveillance camera of an anti-fatigue monitoring system.

[0151] It is worth mentioning that Figure 4 The device provided in the illustrated embodiment can execute the method provided in any of the above method embodiments. Its specific implementation principles, technical features, professional terminology explanations and technical effects are similar and will not be repeated here.

[0152] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 500 may include: at least one processor 501 and a memory 502. Figure 5 An electronic device is shown using a processor as an example.

[0153] The memory 502 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0154] The memory 502 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0155] The processor 501 is configured to execute computer-executable instructions stored in the memory 502 to implement the methods described in the above method embodiments.

[0156] The processor 501 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0157] Optionally, the memory 502 may be independent or integrated with the processor 501. When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0158] Bus 503 is used to connect the processor 501 and the memory 502. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc., but this does not mean that there is only one bus or only one type of bus.

[0159] Optionally, in a specific implementation, if the memory 502 and the processor 501 are integrated on a chip, the memory 502 and the processor 501 can communicate through an internal interface.

[0160] An embodiment of the present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program code. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above-mentioned method embodiments.

[0161] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in the above-mentioned method embodiments when executed by a processor.

[0162] The present invention also provides an anti-fatigue monitoring system, comprising: at least one monitoring camera and Figure 5The electronic device shown; wherein the electronic device is used to calibrate the external parameter conversion matrix of the surveillance camera to implement the methods in the above-mentioned method embodiments.

[0163] An embodiment of the present application also provides a vehicle, comprising the above-mentioned anti-fatigue monitoring system.

[0164] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera extrinsic parameter calibration method, characterized in that: include: When it is detected that the camera to be calibrated fails to perform self-calibration, a calibration prompt message is output, wherein the calibration prompt message is used to instruct the user to perform a preset visual interaction action multiple times on different first calibration extrinsic parameters, where the first calibration extrinsic parameter is outside the current image acquisition range of the camera to be calibrated; Acquiring, through the camera to be calibrated, a calibration image when the user performs the preset visual interaction action; Determining relative position information among the camera to be calibrated, the user, and each of the first calibration external parameters based on the calibration image using an image analysis model; Determine the extrinsic parameter conversion matrix of the camera to be calibrated after recalibration according to the relative position information.

2. The camera extrinsic parameter calibration method according to claim 1, wherein: The method of using an image analysis model to determine relative position information among the camera to be calibrated, the user, and each of the first calibration external parameters according to the calibration image includes: Determining, using the image analysis model and based on the calibration image, first relative position information between the user and the camera to be calibrated, and second relative position information between the user and one of the first calibration extrinsic parameters, the relative position information including: the first relative position information and the second relative position information; Correspondingly, determining the extrinsic parameter conversion matrix of the camera to be calibrated after recalibration according to the relative position information includes: Determining the current position coordinates of the camera to be calibrated in the preset coordinate system according to the position coordinates of each first calibration extrinsic parameter in the preset coordinate system, the first relative position information, and each second relative position information; According to the current position coordinates and the latest historical position coordinates of the camera to be calibrated, the extrinsic parameter conversion matrix is ​​corrected and transformed to recalibrate the extrinsic parameter conversion matrix.

3. The camera extrinsic parameter calibration method according to claim 2, characterized in that: The position of each of the first calibration extrinsic parameters in the preset coordinate system is constant, and the position of the camera to be calibrated in the preset coordinate system is variable.

4. The camera extrinsic parameter calibration method according to claim 2, wherein: The first relative position information includes: a first relative direction and / or a first relative distance between the user and the camera to be calibrated; the second relative position information includes: a second relative direction and / or a second relative distance between the user and the first calibration extrinsic parameter.

5. The camera extrinsic parameter calibration method according to claim 2, wherein: The modifying and transforming the extrinsic parameter conversion matrix according to the current position coordinates and the latest historical position coordinates of the camera to be calibrated to recalibrate the extrinsic parameter conversion matrix includes: Calculating a coordinate offset vector of the current position coordinates and the historical position coordinates; The extrinsic parameter conversion matrix is ​​transformed according to the coordinate offset vector to recalibrate the extrinsic parameter conversion matrix.

6. The camera extrinsic parameter calibration method according to any one of claims 1 to 5, characterized in that: The preset visual interaction action includes: gazing at the first calibration extrinsic parameter.

7. The camera extrinsic parameter calibration method according to claim 6, characterized in that: The gazing at the first calibration extrinsic parameter includes: the user facing forward and turning eyes toward the first calibration extrinsic parameter, or the user facing the first calibration extrinsic parameter and looking directly at the first calibration extrinsic parameter with the eyes.

8. The camera extrinsic parameter calibration method according to claim 6, characterized in that: Each of the first calibration external parameters is provided with at least one of a display, a light emitter, a reflector, and a light diffuser.

9. The camera extrinsic parameter calibration method according to any one of claims 1 to 5, characterized in that: The detection of the failure of the self-calibration of the camera to be calibrated includes: Obtaining the user's current gaze area and interested gaze area, where the current gaze area is determined based on real-time image data collected by the camera to be calibrated, and the first position coordinates of the camera to be calibrated in a preset coordinate system are variable. The interested gaze area is an area where the probability of the user's gaze being observed when performing a preset task multiple times is greater than a preset threshold; Determining whether a deviation between the current gaze area and the interested gaze area is greater than a preset deviation threshold; If so, it is determined that the extrinsic parameter conversion matrix of the camera to be calibrated needs to be calibrated and corrected.

10. The camera extrinsic parameter calibration method according to claim 9, characterized in that: Before obtaining the current gaze area of ​​the user, the method further includes: When the camera to be calibrated is powered on, starting a self-calibration mode of the camera to be calibrated, so that the camera to be calibrated performs self-calibration according to preset features in the collected real-time image; If the self-calibration fails after being repeated N times, the extrinsic parameter conversion matrix of the camera to be calibrated is set to the historical extrinsic parameter conversion matrix obtained when the self-calibration was successful last time.

11. The camera extrinsic parameter calibration method according to claim 1, characterized in that: The camera to be calibrated is a surveillance camera of an anti-fatigue monitoring system.

12. A camera external parameter calibration device, characterized in that: include: The acquisition module is used to obtain the self-calibration status of the camera to be calibrated; Processing module for: When it is detected according to the self-calibration situation that the camera to be calibrated fails to perform self-calibration, outputting calibration prompt information, wherein the calibration prompt information is used to instruct the user to perform a preset visual interaction action multiple times on different first calibration extrinsic parameters, where the first calibration extrinsic parameter is outside the current image acquisition range of the camera to be calibrated; Acquiring, through the camera to be calibrated, a calibration image when the user performs the preset visual interaction action; Determining relative position information among the camera to be calibrated, the user, and each of the first calibration external parameters based on the calibration image using an image analysis model; Determine the extrinsic parameter conversion matrix of the camera to be calibrated after recalibration according to the relative position information.

13. An electronic device, characterized in that: include: processor; as well as, a memory for storing a computer program for the processor; The processor is configured to execute the camera extrinsic parameter calibration method according to any one of claims 1 to 11 by executing the computer program.

14. An anti-fatigue monitoring system, characterized in that: include: at least one surveillance camera and the electronic device according to claim 13; The electronic device is used to calibrate the external parameter conversion matrix of the surveillance camera to implement the camera external parameter calibration method described in any one of claims 1 to 11.

15. A vehicle, characterized in that: Including the anti-fatigue monitoring system according to claim 14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the camera extrinsic parameter calibration method according to any one of claims 1 to 11 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the camera extrinsic parameter calibration method according to any one of claims 1 to 11 is implemented.

Images