Line of sight area detection method, system, device, medium and vehicle

Abstract

The present disclosure relates to a line of sight area detection method, system, device, medium and vehicle, the method comprising: acquiring a monocular image; acquiring a stereoscopic reference model, the stereoscopic reference model being obtained by three-dimensionally reconstructing a series of reference monocular images under different viewing angles; determining a reference monocular image that matches the monocular image based on the monocular image and the stereoscopic reference model; determining the user's head posture based on the matched reference monocular image and the monocular image; and determining the user's line of sight area range based on the head posture. According to the technical solution of the present disclosure, the line of sight area detection method has low computational complexity, which is conducive to improving computational efficiency; at the same time, it can be applied to a variety of different scenarios, is less restricted by scenarios, and has a wider applicability, that is, the application scenario has less influence on the computational results, which is conducive to more accurately determining the user's line of sight area range, that is, it is conducive to accurate detection of the user's line of sight area range.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a line of sight area detection method, system, device, medium and vehicle. Background Art

[0002] With the development of economy and technology, vehicles are increasingly becoming a means of daily travel. During driving, the driver's state, such as whether he is focused or not, is usually closely related to driving safety. One way to monitor the driver's state is to determine whether he is focused based on the driver's field of vision.

[0003] Among them, the detection of the driver's field of vision can be achieved based on the analysis of his head posture. There are two main existing methods for detecting the field of vision. One method is to use a depth sensor device to obtain three-dimensional information, and then estimate the head posture to determine the field of vision; but its device is complex and the data processing process is complicated. At the same time, the current mainstream image acquisition device usually obtains two-dimensional images, resulting in poor universality of this method.

[0004] Another method is to detect, track and predict key feature points by collecting multi-view images or video sequences, and then estimate the head posture and determine the range of the visual area. Its universality is relatively high, but when this method is applied to situations where the range of the visual area changes greatly, its accuracy is poor and its operating efficiency is low. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a line of sight area detection method, system, device, medium and vehicle.

[0006] The present disclosure provides a sight area detection method based on a monocular image, comprising:

[0007] Get a monocular image;

[0008] Acquire a stereo reference model; wherein the stereo reference model is obtained by three-dimensionally reconstructing a series of reference monocular images at different viewing angles;

[0009] Based on the monocular image and the stereo reference model, determining the reference monocular image that matches the monocular image;

[0010] Determining a head posture of a user based on the matched reference monocular image and the monocular image;

[0011] Based on the head posture, the user's visual field range is determined.

[0012] The present disclosure also provides a sight area detection system based on a monocular image, comprising:

[0013] An image acquisition module, used to acquire a monocular image;

[0014] A model acquisition module, used for acquiring a three-dimensional reference model;

[0015] An image matching module, configured to determine the reference monocular image that matches the monocular image based on the monocular image and the stereo reference model;

[0016] A posture determination module, used to determine the user's head posture based on the matched reference monocular image and the monocular image;

[0017] The sight area determination module is used to determine the sight area range of the user based on the head posture.

[0018] The present disclosure also provides an electronic device, comprising:

[0019] memory and one or more processors;

[0020] The memory is communicatively connected to the one or more processors, and the memory stores instructions that can be executed by the one or more processors. When the instructions are executed by the one or more processors, the electronic device is used to implement any of the above methods.

[0021] The present disclosure also provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed by a computing device, they can be used to implement any of the above methods.

[0022] The present disclosure also provides a vehicle that applies any of the above methods to estimate the driver's field of vision based on a monocular image, or includes any of the above systems.

[0023] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0024] The line of sight area detection method, system, device, medium and vehicle provided in the embodiments of the present disclosure obtain a monocular image and a stereo reference model, and determine a reference monocular image in the stereo model that matches the monocular image, and further determine the user's head posture based on the two images, and then determine the user's line of sight area range based on the user's head posture. The calculation complexity is low, which is conducive to improving operation efficiency. At the same time, it can be applied to various different scenarios, has a wide applicability, is less restricted by scenarios, and has little impact on the calculation results by the application scenarios. The accuracy in each scenario is high, which is conducive to realizing accurate detection of the user's line of sight area range. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A schematic diagram of a flow chart of a sight area detection method provided in an embodiment of the present disclosure;

[0028] Figure 2 A schematic diagram of a monocular imaging process provided by an embodiment of the present disclosure;

[0029] Figure 3 A schematic diagram of a 3D standard model provided in an embodiment of the present disclosure;

[0030] Figure 4 A schematic diagram of a head posture angle provided in an embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of the structure of a sight area detection system provided by an embodiment of the present disclosure;

[0032] Figure 6 A schematic diagram of the structure of another sight area detection system provided by an embodiment of the present disclosure;

[0033] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0036] The technical solution provided by the embodiments of the present disclosure can be applied to any scenario where it is necessary to estimate the scope of the user's visual area, including but not limited to human-computer interaction, virtual reality, intelligent driving, etc. In the implementation of the method, monocular image information including the user's head can be collected based on an image acquisition device such as a camera or a camera, and the user's head posture can be estimated in combination with camera parameters, a stereo reference model, etc., and the scope of the user's visual area can be further determined.

[0037] In some embodiments, the solution can be applied to self-driving taxis (Robotaxi), using the on-board monocular camera in the self-driving taxi to obtain a monocular image including the head of the driver (such as a safety officer), and combined with a stereo reference model to determine the driver's head posture, so as to subsequently judge the driver's field of vision and determine whether his attention is focused, thereby enabling monitoring of the driver's status.

[0038] Combine the following Figure 1-Figure 4 , the line of sight area detection method based on monocular image provided by an embodiment of the present disclosure is exemplarily described.

[0039] In some embodiments, Figure 1 A flowchart of a line of sight area detection method provided in an embodiment of the present disclosure.

[0040] In S110 , a monocular image is acquired.

[0041] The monocular image is an image taken by a monocular camera and is a two-dimensional plane image. The monocular image includes the user's head area. The imaging process principle of the monocular image can be found in Figure 2 .

[0042] In this step, an image acquisition module may be used to acquire a two-dimensional plane image.

[0043] In S120 , a three-dimensional reference model is obtained.

[0044] Among them, the stereo reference model can also be called a 3D face standard model, which can be obtained by three-dimensionally reconstructing a series of reference monocular images under different perspectives. The three-dimensional reconstruction algorithm can adopt the SFM algorithm or other three-dimensional reconstruction algorithms known to technical personnel in this field, which is not limited here.

[0045] In S130 , a reference monocular image matching the monocular image is determined based on the monocular image and the stereo reference model.

[0046] The reference monocular image and the monocular image refer to the images having the highest similarity between the two, which are exemplarily described below in conjunction with feature points and feature descriptors.

[0047] Exemplarily, among the reference monocular images at multiple different perspectives of the reconstructed stereo reference model, the reference monocular image with the highest similarity to the monocular image can be determined by comparison, based on the reference monocular image matching.

[0048] In S140 , the user's head posture is determined based on the matched reference monocular image and the monocular image.

[0049] The user's head posture can be represented by the user's head posture angle, and the user's head posture angle can be the rotation angle of the user's head from the reference position in the three-dimensional coordinate system, see Figure 4 .

[0050] In this step, based on the spatial position conversion relationship between the monocular image including the user's head area and the corresponding reference monocular image, the user's head posture angle can be judged, that is, the user's head posture can be determined.

[0051] In S150 , the user's visual area is determined based on the head posture.

[0052] The user's visual area range can represent the direction and degree of the user's attention. Based on the user's head posture, the visual area range can be determined to further determine whether the user's attention is focused and where the user's attention is focused.

[0053] Therefore, in the line of sight area detection method based on monocular images, the reference monocular image matching the monocular image in the stereo model can be determined based on the monocular image and the stereo reference model, and the user's head posture angle can be further judged based on the spatial position conversion relationship between the two images to determine the user's line of sight area range. It is not restricted by the scene and can be applied to a variety of different platforms. It has low computational complexity, high computational efficiency, and high accuracy.

[0054] In other implementations, S120 may be executed before S110, or the two may be executed simultaneously, which is not limited in the embodiments of the present disclosure.

[0055] On the basis of the above implementation, before using the monocular camera to obtain the monocular image, the monocular camera can be calibrated, and the camera parameters of the calibrated monocular camera can be applied to the subsequent steps to ensure the coordinate transformation (combined with Figure 2 Understanding) accuracy, thereby ensuring high accuracy of line of sight area detection.

[0056] Thus, in some embodiments, Figure 1 Based on this, before S110, the method may further include:

[0057] Offline calibration of monocular cameras;

[0058] Get the calibrated camera parameters.

[0059] Specifically, the camera in the monocular camera is calibrated to obtain calibrated camera parameters.

[0060] Exemplarily, the camera parameters may include the intrinsic parameters and distortion parameters of the camera. The intrinsic parameters include but are not limited to the focal length of the camera and the optical center of the image (which may be simply referred to as the "optical center"), and the distortion parameters include but are not limited to the radial distortion coefficient and tangential distortion coefficient of the camera.

[0061] The method of offline calibration of the monocular camera may adopt any method known to those skilled in the art, which will not be elaborated or limited here.

[0062] In the above implementation, after acquiring the monocular image, the facial region information may be determined first, and then the user's facial key feature point information may be determined, and the user's head posture may be further determined in combination with the camera parameters and the stereo reference model.

[0063] In some embodiments, Figure 1 Based on this, S130 may specifically include:

[0064] Based on the monocular image, extract the features and feature descriptors of the monocular image;

[0065] Obtain reference features and reference feature descriptors of a reference monocular image;

[0066] A reference monocular image that matches the monocular image is determined based on the features, the feature descriptors, the reference features, and the reference feature descriptors.

[0067] Among them, one of the methods such as scale-invariant features transform (SIFT), histogram of oriented gradient (HOG), difference of Gaussian (DOG), accelerated robust features (SURF), local binary pattern (LBP) can be used to extract features from the monocular image, so as to achieve more accurate extraction of features in the monocular image.

[0068] For example, the scale-invariant feature transformation method can be used to extract features from monocular images.

[0069] In other implementations, other methods known to those skilled in the art may be used to implement feature extraction based on monocular images, which will not be elaborated or limited herein.

[0070] Among them, the features and reference features include contours, corner points, areas with large contrast changes (for example, contrast changes exceeding a set threshold) and other optional features. The feature descriptors and reference feature descriptors are both binary numbers consisting of a string of 01s, which are used to characterize the features.

[0071] Based on this, the similarity between the monocular image and the reference monocular image can be determined by calculating the weighted sum of all feature descriptors between the monocular image and the reference monocular image, and comparing the relative size of the weighted sum between the two images, wherein the smaller the difference in the weighted sum between the two images, the higher the similarity. The weights of different features can be set based on their influence on the image similarity, and their specific values ​​are not limited here.

[0072] In some embodiments, Figure 1 Based on this, S140 may specifically include:

[0073] Based on the monocular image, determine the user's facial area information;

[0074] Locate the key facial feature points of the user based on the facial region information and determine the key facial feature point information;

[0075] Based on the reference monocular image matched with the monocular image and the facial key feature point information, determine the three-dimensional point information corresponding to the facial key feature points one by one;

[0076] Based on the camera parameters, the information of key facial feature points and the corresponding three-dimensional point information, the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system is solved;

[0077] Determine the user's head posture based on the projection transformation relationship;

[0078] Wherein, the shape of the facial region is rectangular, circular or oval;

[0079] The facial parts corresponding to the key feature point information include the corners of the eyes, the corners of the mouth, the tip of the nose, the left and right ends of the nose wings, the tip of the chin, and the facial contour;

[0080] The camera parameters include the focal length of the monocular camera in the X and Y directions, the position of the optical center of the lens in the monocular camera in the X and Y directions, and the scale parameters during coordinate conversion.

[0081] Specifically, face detection can be performed on a monocular image to obtain user's facial region information. For example, the facial region can be represented by a rectangular frame, or other shapes such as a circle or an ellipse can be used to represent the facial region.

[0082] Among them, face detection can be implemented based on a face detection algorithm. The face detection algorithm refers to using a face classifier to detect the user's face in a monocular image, including but not limited to using traditional vision algorithms, deep learning algorithms or other algorithms known to technical personnel in this field.

[0083] Among them, the facial area information refers to the two-dimensional coordinate information of the user's facial area in the current monocular image coordinate system, including the coordinates of the upper left corner vertex and length and width information of the rectangular frame, or the center and radius information of the circle, as well as positioning information and characterization information of other shapes.

[0084] Based on the facial area information, visual algorithms or deep learning algorithms can be used to automatically locate the key facial feature points, that is, their position information.

[0085] The user's facial key feature point information may be the coordinate position of the user's facial key feature points in a monocular image, which may be represented by the coordinate position in a two-dimensional rectangular coordinate system. The user's facial key feature points are used to characterize the user's head posture, including but not limited to the user's facial contour points, facial features contour points, and other convex and concave feature points.

[0086] In this step, a visual algorithm or a deep learning algorithm known to those skilled in the art may be used to determine the user's facial key feature point information based on a monocular image.

[0087] For example, the number of key facial feature points of a user may be 68, see Figure 3 In other embodiments, the number of key facial feature points of the user may be set to be more to improve the estimation accuracy of the head posture; it may also be set to be less to reduce the computational complexity and improve the computational efficiency, and may be set based on the requirements of the sight area detection method and is not limited here.

[0088] On this basis, the user's head posture can be determined by combining the camera coordinate system and solving the projection transformation relationship between the world coordinate system and the image coordinate system.

[0089] Specifically, by comparing the facial key feature point information with the reference monocular image, the three-dimensional point information corresponding one-to-one to the facial key feature points can be determined, that is, a feature point set corresponding to the user's facial key feature points in the stereo reference model is obtained, which is a three-dimensional point set determined based on the stereo reference model and the reference monocular image.

[0090] The conversion between the feature point set and the facial key feature point information is realized based on the projection transformation relationship, which can be combined with Figure 2Based on this, the projection transformation relationship represents the relationship between the world coordinate system, the camera coordinate system and the image coordinate system for coordinate transformation, and the angle change included in it can be used to represent the user's head posture, thereby realizing the estimation of the user's head posture.

[0091] In other embodiments, other methods known to those skilled in the art may be used to determine the user's facial key feature point information based on a monocular image. Based on this, S130 may include:

[0092] Based on the monocular image, determine the user's facial key feature point information;

[0093] Based on the reference monocular image matched with the monocular image and the facial key feature point information, determine the three-dimensional point information corresponding to the facial key feature points one by one;

[0094] Based on the camera parameters, the information of key facial feature points and the corresponding three-dimensional point information, the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system is solved;

[0095] Based on the projection transformation relationship, the user's head posture is determined.

[0096] In some embodiments, Figure 1 Based on the combination Figure 2 In the above S130, “solving the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system” may include:

[0097] Based on Formula 1 and Formula 2, solve the projection transformation relationship between the world coordinate system, camera coordinate system and image coordinate system:

[0098]

[0099]

[0100] Among them, (x, y) represents the key feature point information of the face, (X c ,Y c ,Z c ) represents the position of the corresponding point in the monocular camera coordinate system, (X w ,Y w ,Z w ) represents the corresponding three-dimensional point information, R represents the rotation matrix, t represents the translation vector, f x and f y Represents the focal length of the monocular camera in the X and Y directions, c x and c y They represent the positions of the optical center of the lens in the monocular camera in the X and Y directions respectively, and s represents the scale parameter during coordinate transformation;

[0101] Among them, the camera parameters include f x 、f y 、c x 、c y and s;

[0102] The user's head posture is the Euler angle determined based on the rotation matrix R, including the pitch angle, yaw angle, and roll angle.

[0103] Among them, by setting the camera parameters in the calculation process, including the focal length of the monocular camera in the X direction and the Y direction, the position of the lens optical center in the monocular camera in the X direction and the Y direction, and the scale parameters during coordinate conversion, the camera performance can be combined to more accurately restore the user's head posture based on the monocular image, thereby improving the calculation accuracy of the user's head posture.

[0104] In other embodiments, in order to further reduce the impact of camera performance on coordinate transformation and improve calculation accuracy, the camera parameters may also be set to include other parameters that affect the image effect of the monocular image, such as radial distortion coefficient and tangential distortion coefficient, etc., which are not limited here.

[0105] Combination Figure 4 The pitch angle represents the angle of rotation around the X-axis and is represented by pitch; the yaw angle, also known as the roll angle, represents the angle of rotation around the Y-axis and is represented by yaw; the roll angle represents the angle of rotation around the Z-axis and is represented by roll.

[0106] For example, taking the intelligent driving scenario as an example, the field of vision area includes the front, upper left, lower left, upper right, and lower right of the front of the vehicle.

[0107] In a series of embodiments, the front of the sight area refers to a pitch range between -20° (upper) and 20° (lower) angles, and a yaw range between -20° (left) and 30° (right) angles; the upper left of the sight area refers to a pitch range less than -20°, and a yaw range less than -20°; the lower left of the sight area refers to a pitch range greater than 20°, and a yaw range less than -20°; the upper right of the sight area refers to a pitch range less than -20°, and a yaw range greater than 30°; the lower right of the sight area refers to a pitch range greater than 20°, and a yaw range greater than 30°.

[0108] In other implementations, the angle value of the sight area may also be set according to changes in the actual environment inside and outside the vehicle.

[0109] In other implementations, when other coordinate systems are used to implement coordinate conversion, other methods may also be used to represent the user's head posture and visual area range, which will not be elaborated or limited here.

[0110] In some embodiments, solving the projection transformation relationship between the world coordinate system, the camera coordinate system, and the image coordinate system includes:

[0111] The direct linear transformation method is used to solve the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system based on Formula 1 and Formula 2.

[0112] Among them, the direct linear transformation method is used to solve the projection transformation relationship. The solution method is simple, the calculation difficulty is small, and the calculation efficiency is high.

[0113] In other implementations, other transformation methods may be used to solve the projection transformation relationship to further improve the calculation accuracy, thereby improving the accuracy of the sight area detection method.

[0114] In some embodiments, Figure 1 On the basis of the above, the method may also include: using a display module to display the user's head posture or visual area range determined above, and further providing an early warning based on the above, so as to improve safety. For example, when applied to intelligent driving scenarios, at least one of the driver's head posture and visual area range is displayed to determine the driver's state; and an early warning is provided when the driver is not paying attention or makes an unexpected operation.

[0115] Therefore, in Figure 1 On the basis of, after S140, the method may further include:

[0116] Display at least one of a head posture and a visual area range;

[0117] Based on at least one of the head posture and the visual area range, the warning prompt information is determined.

[0118] Specifically, when the head posture and the sight area range are displayed simultaneously, the two are updated synchronously in real time during display to ensure the real-time and synchronization of the display, thereby ensuring the accuracy of the detection of the user's head posture and the sight area range.

[0119] The disclosed embodiment also provides a line of sight area detection system based on a monocular image, which is used to execute the steps of any of the above methods to achieve the corresponding effects.

[0120] In some embodiments, Figure 5 A schematic diagram of the structure of a sight area detection system provided in an embodiment of the present disclosure.

[0121] like Figure 5As shown, the sight area detection system based on monocular image includes: an image acquisition module 210, used to acquire a monocular image; a model acquisition module 220, used to acquire a stereo reference model; an image matching module 230, used to determine a reference monocular image that matches the monocular image based on the monocular image and the stereo reference model; a posture determination module 240, used to determine the user's head posture based on the matched reference monocular image and the monocular image; a sight area determination module 250, used to determine the user's sight area range based on the head posture.

[0122] Therefore, in the line of sight area detection system based on monocular image, through the synergistic effect of the above-mentioned functional modules, based on the acquired monocular image and the stereo reference model, a reference monocular image matching the monocular image can be determined, the user's head posture angle can be judged based on the matched reference monocular image and the monocular image, and the user's line of sight area range can be further determined. It is not restricted by the scene and can be applied to a variety of different platforms. It has low computational complexity, high computational efficiency, and high accuracy.

[0123] In some embodiments, when the system is applied in a vehicle, the image acquisition module may include an on-board monocular camera, such as an on-board camera module; correspondingly, the user includes a driver.

[0124] Therefore, the system can estimate the driver's head posture and visual area range, realize the driver's posture monitoring, and help improve driving safety.

[0125] In some embodiments, the system may include:

[0126] The image acquisition module 240 may be a vehicle-mounted camera, and its hardware implementation may be a vehicle-mounted camera, which is installed in a horizontal position above the dashboard in the car to collect image information of the driver.

[0127] Among them, the best installation position for the vehicle-mounted camera is to obtain a greater degree of driver's facial information without affecting the driver's operation.

[0128] In some embodiments, the vehicle-mounted camera needs to be calibrated offline. Based on this, a camera offline calibration module is set to calibrate the vehicle-mounted camera to obtain camera parameters.

[0129] In some embodiments, the image matching module may include:

[0130] The monocular image feature extraction submodule is used to extract the features and feature descriptors of the monocular image based on the monocular image; the reference monocular image feature acquisition submodule is used to obtain the reference features and reference feature descriptors of the reference monocular image; the image matching submodule is used to determine the reference monocular image that matches the monocular image based on the features, feature descriptors, reference features and reference feature descriptors.

[0131] In this way, the comparison between the images is achieved, and a reference monocular image matching the monocular image is determined.

[0132] In some embodiments, the posture determination module may specifically include a face detection module, a facial shape positioning module, a related point set determination module and a head posture estimation module.

[0133] The face detection module can perform face detection based on the monocular image to obtain the driver's facial area information. Specifically, the face detection module can be implemented based on a face detection algorithm, including using a face classifier to perform driver detection on the monocular image information to obtain the driver's facial area information.

[0134] The facial shape positioning module can automatically locate the position information of the facial significant feature points according to the facial area information determined by the face detection module, that is, obtain the facial key feature point information. Specifically, the facial significant feature points include the positions of the key components that make up the face, such as the eyes, the corners of the mouth, the tip of the nose, and the facial contour.

[0135] Among them, the positioning of facial key feature points can be regarded as a regression problem, including but not limited to using traditional vision or deep learning algorithms for solving it.

[0136] The associated point set determination module can determine the three-dimensional point information corresponding to the facial key feature points one by one based on the reference monocular image matched with the monocular image and the facial key feature point information.

[0137] The head posture estimation module can determine the driver's head posture based on the projection transformation relationship between different coordinate systems and the calibrated camera parameters during the target imaging process, and solve the transformation relationship matrix between the reference monocular image and the corresponding point set between the monocular image based on the direct linear transformation algorithm. The above-mentioned different coordinate systems include the world coordinate system, the camera coordinate system, and the image coordinate system. The projection transformation relationship is as follows: Figure 2 As shown, the solution method can be understood by referring to the above text and will not be repeated here.

[0138] In some embodiments, Figure 6 This is a schematic diagram of the structure of another sight area detection system provided by an embodiment of the present disclosure. Figure 6 As shown, the system may also include:

[0139] The display module 260 is used to display at least one of the head posture and the visual area range; the prompt information determination module 270 is used to determine the warning prompt information based on at least one of the head posture and the visual area range.

[0140] Specifically, the display module 260 may be a 3D visual display module, which is used to display the 3D face shape corresponding to the current frame monocular image in real time, including at least one of its head posture and sight area range, so as to achieve real-time synchronization update of the 3D face shape corresponding to the current frame monocular image on the display interface. . The 3D face shape refers to the three-dimensional coordinate point set obtained after the 3D face standard model is transformed by the posture angle rotation.

[0141] For example, in combination Figure 4 , the 3D face shape is based on the 3D face standard model, and each posture angle in the 3D face standard model is 0°. Correspondingly, the range of the user's posture angle corresponds to the 3D face shape, the pitch angle ranges from 0° to -90° for upward looking, and from 0° to 90° for downward looking; the roll angle ranges from 0° to -90° for rolling to the left, and from 0° to 90° for rolling to the right; the yaw angle ranges from 0° to 90° for deflection to the left, and from 0° to -90° for deflection to the right.

[0142] In some embodiments, a safety threshold is built into the prompt information determination module 270. By comparing the user's field of vision with the safety threshold, the driver's attention direction is determined, thereby determining whether the user's attention is focused on driving to determine whether his current state is safe; and when the user's current state corresponds to an unsafe state, early warning prompt information is generated to implement safety early warning prompts and improve safety.

[0143] So, based on Figure 6 The system shown can determine the driver's head posture and visual area range with high operating efficiency and accuracy.

[0144] Figure 7 It is a structural diagram of an electronic device suitable for implementing an embodiment of the present disclosure.

[0145] like Figure 7 As shown, the electronic device 500 includes a central processing unit (CPU) 501, which can execute the above-mentioned programs according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. Figure 1-3Various processes in the illustrated embodiment. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0146] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0147] In particular, according to the implementation of the embodiment of the present disclosure, the above reference Figure 1 The described method may be implemented as a computer software program. For example, an implementation of the present disclosure includes a computer program product, which includes a computer program tangibly embodied on a medium readable by the computer, the computer program including a computer program for executing Figure 1 In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 509 , and / or installed from the removable medium 511 .

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the road map or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The units or modules involved in the implementation of the embodiments of the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0150] As another aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0151] As another aspect, the disclosed embodiment also provides a vehicle, which can apply any of the methods in the above embodiments to estimate the driver's visual area based on a monocular image; correspondingly, the vehicle can include any of the above systems. Specifically, first, the vehicle camera module is offline calibrated to obtain the parameters of the camera; secondly, the driver's image information is obtained through the camera to obtain a monocular image; a stereo reference model is obtained, and a reference monocular image matching the monocular image in the reference monocular image used to reconstruct the stereo reference model is determined; thereafter, the driver's facial area information is detected, and the face shape information is located on this basis; then, the transformation relationship matrix is ​​solved based on the projection transformation and direct linear transformation algorithm between different coordinate systems in the image imaging process to determine the driver's head posture angle; finally, the 3D face shape and posture corresponding to the current frame monocular image are obtained by coordinate transformation calculation, and visual display is performed; based on the current posture angle analysis, the driver's visual area range is judged and extracted. In this way, the estimation of the driver's visual area range is realized, and its operation efficiency and accuracy are high; at the same time, the monocular image is obtained by the vehicle-mounted camera, and there is no need to install an image sensor on the driver's body, and the use method is simple and convenient.

[0152] In summary, the embodiments of the present disclosure propose a sight area detection method, system, electronic device, computer readable storage medium thereof, and vehicle. By performing sight area detection based on monocular images and stereo reference models, the computational complexity is low and the operation efficiency is high; at the same time, it is applicable to various scenarios, has a wide range of applicability, is less restricted by scenarios, has less impact on the computational results by the application scenarios, and has high accuracy in all scenarios.

[0153] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0154] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sight area detection method based on a monocular image, characterized in that: include: Get a monocular image; Before obtaining the monocular image, the method further includes: offline calibrating the monocular camera; obtaining the calibrated camera parameters; Acquire a stereo reference model; wherein the stereo reference model is obtained by three-dimensionally reconstructing a series of reference monocular images at different viewing angles; Based on the monocular image and the stereo reference model, determining the reference monocular image that matches the monocular image; Determining a head posture of a user based on the matched reference monocular image and the monocular image; Based on the head posture, determining the user's visual area range; Determining a head posture of a user based on the reference monocular image and the monocular image includes: Based on the monocular image, determine the key feature point information of the user's face; Determine three-dimensional point information corresponding to the facial key feature points one by one based on the reference monocular image matched with the monocular image and the facial key feature point information; Based on the camera parameters, the facial key feature point information and the three-dimensional point information corresponding thereto, solving the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system; Based on the projection transformation relationship, determining the user's head posture; The camera parameters include the focal length of the monocular camera in the X direction and the Y direction, the position of the optical center of the lens in the monocular camera in the X direction and the Y direction, and the scale parameters during coordinate conversion.

2. The method according to claim 1, characterized in that The step of determining the reference monocular image that matches the monocular image based on the monocular image and the stereo reference model comprises: Based on the monocular image, extract the features and feature descriptors of the monocular image; wherein the features in the monocular image are extracted using a scale-invariant feature transformation method, a directional gradient histogram method, a Gaussian function difference method, an accelerated robust feature method or a local binary pattern method; Acquire reference features and reference feature descriptors of the reference monocular image; Determining the reference monocular image that matches the monocular image based on the feature, the feature descriptor, the reference feature, and the reference feature descriptor; The features and the reference features correspond to contours, corner points, and areas where contrast changes exceed a set threshold.

3. The method according to claim 1, characterized in that The method of solving the projection transformation relationship among the world coordinate system, the camera coordinate system and the image coordinate system includes: Based on Formula 1 and Formula 2, solve the projection transformation relationship between the world coordinate system, camera coordinate system and image coordinate system: Formula 1 Formula 2 in, Represents the key feature point information of the face, Represents the position of the corresponding point in the monocular camera coordinate system, represents the corresponding three-dimensional point information, R represents the rotation matrix, t represents the translation vector, and f x and f y Represents the focal length of the monocular camera in the X and Y directions, c x and c y They represent the positions of the optical center of the lens in the monocular camera in the X and Y directions respectively, and s represents the scale parameter during coordinate transformation; The camera parameters include f x 、f y 、c x 、c y and s; The user's head posture is the Euler angle determined based on the rotation matrix R, including the pitch angle, the yaw angle and the roll angle.

4. The method according to claim 1, characterized in that: After determining the user's visual area based on the head posture, the method further includes: Display at least one of the head posture and the visual area range; Based on at least one of the head posture and the visual area range, early warning prompt information is determined.

5. A sight area detection system based on monocular image, characterized in that: include: An image acquisition module, used to acquire a monocular image; Before obtaining the monocular image, the method further includes: offline calibrating the monocular camera; obtaining the calibrated camera parameters; A model acquisition module is used to acquire a stereo reference model; wherein the stereo reference model is obtained by three-dimensionally reconstructing a series of reference monocular images at different viewing angles; An image matching module, configured to determine the reference monocular image that matches the monocular image based on the monocular image and the stereo reference model; a posture determination module, configured to determine the user's head posture based on the matched reference monocular image and the monocular image; A sight area determination module, used to determine the sight area range of the user based on the head posture; Wherein, determining the head posture of the user based on the reference monocular image and the monocular image includes: Based on the monocular image, determine the key feature point information of the user's face; Determine three-dimensional point information corresponding to the facial key feature points one by one based on the reference monocular image matched with the monocular image and the facial key feature point information; Based on the camera parameters, the facial key feature point information and the three-dimensional point information corresponding thereto, solving the projection transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system; Based on the projection transformation relationship, determining the user's head posture; The camera parameters include the focal length of the monocular camera in the X direction and the Y direction, the position of the optical center of the lens in the monocular camera in the X direction and the Y direction, and the scale parameters during coordinate conversion.

6. The system according to claim 5, characterized in that Also includes: A display module, configured to display at least one of the head posture and the visual area range; The prompt information determination module is used to determine the warning prompt information based on at least one of the head posture and the visual area range.

7. An electronic device, characterized in that: include: memory and one or more processors; The memory is communicatively connected to the one or more processors, and the memory stores instructions that can be executed by the one or more processors. When the instructions are executed by the one or more processors, the electronic device is used to implement the method as described in any one of claims 1-4.

8. A computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a computing device, can be used to implement the method according to any one of claims 1 to 4.

9. A vehicle, characterized in that: Comprising the system of claim 5 or 6.

Images