In-vehicle child monitoring method and system based on millimeter wave radar and vision fusion

Abstract

The invention provides an in-vehicle child monitoring method and system based on millimeter-wave radar and vision fusion, and the method comprises the steps: periodically obtaining millimeter-wave radar data and infrared images in an area where a child is located in a driving state; projecting target information in the millimeter wave radar data into the infrared image, and determining a focus region of interest; a detection area is obtained through a frame difference method based on a key concern area determined by the adjacent frame infrared images; based on the image of the detection area, obtaining a dangerous behavior judgment result through pre-training a child dangerous behavior judgment model based on a neural network; and when the dangerous behavior occurs, giving an alarm and carrying out real-time panoramic display on the child area until the dangerous behavior is finished.

Description

Technical Field

[0001] The present invention belongs to the technical field related to intelligent driving assistance, and in particular relates to a method and system for in-vehicle child monitoring based on millimeter-wave radar and vision fusion. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of intelligent driving assistance functions, driving assistance functions have become standard features of almost all new models of various vehicle manufacturers.

[0004] The inventors found that at present, most child detection systems are more concerned about whether children are left behind in the car, but ignore the function of monitoring children during driving; because children have poor self-control ability, and drivers need to pay attention to the information around the vehicle at all times, it is difficult to effectively perceive the status information of children during the driving process, which can easily lead to safety hazards for children. Summary of the invention

[0005] The present invention provides a method and system for in-vehicle child monitoring based on millimeter-wave radar and vision fusion, so as to solve the problem that traditional solutions cannot effectively monitor children during driving.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for monitoring children in a vehicle based on millimeter wave radar and vision fusion, comprising:

[0008] When the vehicle is in motion, the millimeter-wave radar data and infrared images of the area where the child is located are periodically acquired;

[0009] Project the target information in the millimeter-wave radar data into the infrared image to determine the key areas of interest;

[0010] Based on the focus area determined by the adjacent frame infrared images, the detection area is obtained by using the frame difference method;

[0011] Based on the image of the detection area, the dangerous behavior judgment result is obtained by pre-training a child dangerous behavior judgment model based on a neural network;

[0012] When dangerous behavior occurs, an alarm is issued and a real-time panoramic display of the children's area is provided until the dangerous behavior ends.

[0013] Furthermore, the target information in the millimeter-wave radar data is projected into the infrared image to determine the key focus area, specifically: by jointly calibrating the millimeter-wave radar and the camera, the spatiotemporal synchronization of the radar and image data is achieved, the target information position detected by the radar is projected into the image, the spatiotemporal fusion of the millimeter-wave radar data and the infrared image is achieved, and the key focus area is determined based on the fusion result.

[0014] Furthermore, the spatiotemporal fusion is specifically as follows: based on the positional relationship between the millimeter radar and the camera, and the conversion relationship between the millimeter radar coordinate system and the world coordinate system, a projection matrix of the target information in the millimeter wave radar data to the infrared image is obtained; based on the obtained projection matrix, the spatial fusion of the millimeter wave radar data and the infrared image is realized; and, taking the data collected by the device with the lowest sampling frequency among the camera and the millimeter radar as a benchmark, at the acquisition time corresponding to the benchmark data, the data collected by another device is obtained, and the temporal fusion of the millimeter wave radar data and the infrared image is performed.

[0015] Furthermore, the dangerous behavior judgment result of the child is obtained by pre-training the child dangerous behavior judgment model based on a neural network, specifically by extracting features from the image of the detection area, using the extracted features as input of the child dangerous behavior judgment model, and obtaining the child dangerous behavior judgment result.

[0016] Furthermore, the driving state is determined as follows: when the door signal is turned off and the vehicle is started, it is determined to be in the driving state; when the door signal is turned off and the vehicle is powered off, it is determined to be in the locked state.

[0017] Furthermore, for the non-driving state, when the car is locked, the heart rate and breathing rate in the car are detected by millimeter wave radar; when the breathing rate is detected to be within the preset first range and the heart rate is within the preset second frequency range, it is determined that there is a child in the car and an alarm is issued.

[0018] Furthermore, the alarm adopts a cascade alarm strategy, specifically: when the car locking time is within a first preset time range, the car lights flash and the horn sounds an alarm; when the car locking time is within a second preset time range, an alarm is issued via text message or Internet phone call; when the car locking time exceeds the second preset range, the car lights flash and the horn sounds an alarm, an alarm is issued via text message or Internet phone call, and an alarm is issued.

[0019] In a second aspect, the present invention provides an in-vehicle child monitoring system based on millimeter-wave radar and vision fusion, comprising:

[0020] A data acquisition unit, which is used to periodically acquire millimeter-wave radar data and infrared images of the area where the child is located when the vehicle is in motion;

[0021] A key focus area acquisition unit, which is used to project target information in the millimeter wave radar data into the infrared image to determine the key focus area;

[0022] A detection area acquisition unit, which is used to obtain a detection area by a frame difference method based on the focus area determined by the adjacent frame infrared images;

[0023] A dangerous behavior judgment unit, which is used to obtain a dangerous behavior judgment result by pre-training a child dangerous behavior judgment model based on a neural network based on the image of the detection area;

[0024] The alarm unit is used to issue an alarm when dangerous behavior occurs and to provide a real-time panoramic display of the children's area until the dangerous behavior ends.

[0025] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the method for monitoring children in a vehicle based on the fusion of millimeter-wave radar and vision is implemented.

[0026] According to a fourth aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the in-vehicle child monitoring method based on millimeter-wave radar and vision fusion is implemented.

[0027] One or more of the above technical solutions have the following beneficial effects:

[0028] The present invention provides a method and system for monitoring children in a car based on the fusion of millimeter-wave radar and vision. The scheme uses the fusion of millimeter-wave radar and vision to monitor dangerous behaviors of children in the car while the car is driving, and timely alarms are given when dangerous behaviors are detected, and a panoramic display of the children's area is performed on the center console, which can effectively ensure the safety of children in the driving state, and also prevent the driver from being distracted by paying attention to the children; at the same time, through the cascade alarm strategy, the risk of suffocation of children caused by children still in the car when the car is locked is effectively solved;

[0029] The scheme calibrates the key areas of the collected infrared images through millimeter wave radar, which can effectively determine the areas that need to be focused on, so that the child risk behavior discrimination model can pay more attention to the key differences in the image and improve the accuracy of risk behavior discrimination.

[0030] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a flow chart of a method for in-vehicle child monitoring based on millimeter-wave radar and vision fusion according to an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of the radar and visual three-dimensional space coordinate fusion process according to an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of a child monitoring process in a vehicle while driving according to an embodiment of the present invention;

[0035] FIG4( a ) is a schematic diagram of the overall structure of a child risk behavior judgment model according to an embodiment of the present invention;

[0036] FIG4( b ) is a schematic diagram of the BP neural network structure according to an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of time synchronization between radar data and camera image frames according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0039] It should be noted that a method and system for in-vehicle child monitoring based on millimeter-wave radar and vision fusion, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0040] In one or more embodiments, Figure 1 As shown, the process of monitoring children in a locked car and a driving car is shown. Specifically, this embodiment provides a method for monitoring children in a car based on millimeter wave radar and vision fusion, including:

[0041] Step 1: When driving, periodically obtain millimeter-wave radar data and infrared images of the area where the child is located;

[0042] In a specific implementation, the driving state is determined as follows: when the door signal is turned off and the vehicle is started, it is determined to be in the driving state; when the door signal is turned off and the vehicle is powered off, it is determined to be in the locked state.

[0043] Furthermore, the solution connects the millimeter-wave radar and the infrared camera to the vehicle computer to obtain millimeter-wave radar data and infrared images, and then monitors the children in the vehicle based on the obtained data.

[0044] Step 2: Project the target information in the millimeter-wave radar data into the infrared image to determine the key focus area;

[0045] In a specific implementation, further, the target information in the millimeter-wave radar data is projected into the infrared image to determine the key focus area, specifically: by jointly calibrating the millimeter-wave radar and the camera, the spatiotemporal synchronization of the radar and image data is achieved, the target information position detected by the radar is projected into the image, the spatiotemporal fusion of the millimeter-wave radar data and the infrared image is achieved, and the key focus area is determined based on the fusion result.

[0046] In a specific implementation, the spatiotemporal fusion is specifically as follows: based on the positional relationship between the millimeter radar and the camera, and the conversion relationship between the millimeter radar coordinate system and the world coordinate system, a projection matrix of the target information in the millimeter wave radar data to the infrared image is obtained; based on the obtained projection matrix, the spatial fusion of the millimeter wave radar data and the infrared image is realized; and, taking the data collected by the device with the lowest sampling frequency among the camera and the millimeter radar as a benchmark, at the acquisition time corresponding to the benchmark data, the data collected by another device is obtained, and the temporal fusion of the millimeter wave radar data and the infrared image is performed.

[0047] Specifically, the spatial fusion is as follows:

[0048] By spatially fusing the infrared camera with the millimeter-wave radar, the millimeter-wave radar data and the infrared camera image can be fused into a hybrid image, that is, the information obtained by the radar is mapped to the infrared camera image, and the radar detection data is also used as the key area of ​​the infrared image. Figure 2 shown.

[0049] The optical center of the camera lens is assumed to be the origin, and the X-axis is perpendicular to the ground and downward. c , the Y axis is perpendicular to the body axis and to the right is Y c , Z axis is the direction of the car's forward movement c , that is, the coordinate system is O c -X c Y c Z c Assume that the radar geometric center is the origin and the X-axis is the direction opposite to the radar pin. r , Y axis vertical pin upward Y r , the Z axis is perpendicular to the X and Y axes. r , that is, O r -X r Y r Zr The world coordinate system is O w -X w Y w Z w , the image coordinate system of the infrared camera is O p -X p Y p Z p .

[0050] The conversion relationship between the radar coordinate system and the world coordinate system is:

[0051]

[0052] Among them, H is the vertical height of the origin of the world coordinate system above the ground, I is O w In X r The offset in the direction, L is O r In Z r The offset of the direction.

[0053] Based on the positional relationship between the radar and the camera and other conditions, the conversion relationship between the world coordinate system and the image coordinate system is obtained as follows:

[0054]

[0055] Among them, (X p , Y p ) is the projection coordinate in the image coordinate system; d x d y are the physical sizes of each pixel in the X-axis and Y-axis directions respectively; (X p0 , Y p0 ) is the camera's principal point offset; f is the camera's focal length; R is the camera's external parameter rotation matrix, which is a 3×3 orthogonal unit matrix; t is the camera's external parameter translation vector; M is the projection matrix. x d y 、(X p0 , Y p0 ), focal length f is the internal parameter of the camera, R and t are the external parameters of the camera, which can be obtained offline through the camera calibration method.

[0056] The time fusion is specifically as follows:

[0057] Time fusion is mainly to synchronize the millimeter-wave radar and infrared camera. Because the acquisition frequency of the millimeter-wave radar is 20Hz, and the camera is 10Hz, the data collected by the millimeter-wave radar and the camera are not the information at the same time, resulting in a time deviation of the data. Therefore, in order to synchronize the camera data with a low sampling frequency as the benchmark, a multi-threaded synchronization method is used to achieve data time synchronization. At time nodes such as 20ms, 40ms, and 60ms, when the camera receives an image frame each time, the radar data corresponding to the current time of the image is obtained, such as Figure 5 , a specific example is shown.

[0058] Step 3: Based on the focus area determined by the adjacent frame infrared images, the detection area is obtained by using the frame difference method;

[0059] Step 4: Based on the image of the detection area, a child dangerous behavior judgment model based on a neural network is pre-trained to obtain a dangerous behavior judgment result; FIG4(a) shows a schematic diagram of the structure of the child dangerous behavior judgment model;

[0060] In a specific implementation, the dangerous behavior judgment result of a child is obtained by pre-training a child dangerous behavior judgment model based on a neural network, specifically by extracting features from the image of the detection area, and using the extracted features as input of the child dangerous behavior judgment model to obtain the child dangerous behavior judgment result.

[0061] In the specific implementation, the model needs to be trained in advance, that is, multiple groups of images of children's dangerous behaviors (children sticking their heads and hands out of the window, children fighting in the car, children using knives in the car, etc.) are collected as training sets, and the image feature values ​​are extracted through the convolutional neural network. The image feature values ​​are input into the BP neural network for training. Figure 4(b) shows the structural diagram of the BP neural network.

[0062] In actual application, the image fused by radar and vision is input into the BP neural network, and the BP neural network classifies it. When images of dangerous behaviors for a certain period of time appear in the classification results, it is judged that the children in the car have dangerous behaviors. At this time, the central control will display the area where the children are located in real time and alarm to remind.

[0063] Step 5: When dangerous behavior occurs, an alarm is issued and a real-time panoramic display of the children's area is provided until the dangerous behavior ends.

[0064] like Figure 3 The figure mainly shows the child monitoring solution in the car while driving.

[0065] In one or more embodiments, for a non-driving state, when the car is locked, the heart rate and breathing rate in the car are detected by millimeter wave radar; when the breathing rate is detected to be within a preset first frequency range and the heart rate is within a preset second frequency range, it is determined that there is a child in the car and an alarm is issued.

[0066] In the specific implementation, the alarm adopts a cascade alarm strategy, specifically: when the car locking time is within the first preset time range, the car lights flash and the horn sounds an alarm; when the car locking time is within the second preset time range, the alarm is issued via text message or Internet phone; when the car locking time exceeds the second preset range, the car lights flash and the horn sounds an alarm, the alarm is issued via text message or Internet phone, and the alarm is issued.

[0067] Specifically, by monitoring the door signal, when the door signal is turned off and the vehicle is powered off, the vehicle is locked. The millimeter wave radar is used to detect the interior of the vehicle. The heart rate of an adult is between 1Hz and 1.6Hz, the breathing rate is between 0.2Hz and 0.4Hz, and the heart rate of a child is between 1.7Hz and 2.2Hz, and the breathing rate is between 0.5Hz and 0.7Hz. When the heart rate and breathing rate of a child are detected when the vehicle is locked, it is determined that there is a child in the vehicle, and a cascade alarm is issued, that is, level 1: 10 seconds after the vehicle is locked, the lights flash and the car horn sounds an alarm; level 2: 5 minutes after the vehicle is locked, if it is determined that there is still a child in the vehicle, the owner and associated owners are continuously reminded by message or phone through the Internet or SMS; level 3: 10 minutes after the vehicle is locked, while the first and second level alarms continue, an alarm is issued through the Internet or SMS. The alarm is released until the door is opened.

[0068] The following is a detailed description of child detection in the locked car state through a specific example:

[0069] (1) Use millimeter-wave radar to send signals to living targets and receive target and noise echo signals every 50 ms;

[0070] (2) Perform a 1D range FFT (Fast Fourier Transform) on the received echo signal to obtain 32 signals divided by distance. Then perform a 2D Doppler FFT (Fast Fourier Transform) on the echo signal to obtain the target's velocity and angle information.

[0071] (3) Perform CFAR detection on the echo signal by filtering out noise and clutter signals through windowing, and send the signal after CFAR detection in 8-bit form to a computer for phase analysis and restoration of the signal waveform;

[0072] (4) Perform spectrum analysis on the restored waveform and convert the phase-time spectrum into a frequency-time spectrum through a digital filter and FFT;

[0073] (5) removing other interference signals by windowing and phasor mean cancellation algorithm, separating the breathing and heartbeat signals from the signal by wavelet separation algorithm, reconstructing the waveform of the separated signal, and estimating the frequency of the breathing and heartbeat signals;

[0074] (6) Compare the normal human respiratory and heart rate range with the detected signal frequency range: the heart rate of an adult is 1 Hz to 1.6 Hz, the respiratory rate is 0.2 Hz to 0.4 Hz, the heart rate of a child is 1.7 Hz to 2.2 Hz, the respiratory rate is 0.5 Hz to 0.7 Hz, and the information is stored in the system;

[0075] (7) After the vehicle is locked, detection begins to determine whether a child is left in the vehicle by judging whether the signal matches the child's heartbeat and breathing rates;

[0076] (8) If the signal is judged to be consistent with the heart rate and breathing rate of a child, a child is left in the vehicle and a graded alarm is issued;

[0077] (9) If the signal is judged not to match the child's heart rate and breathing rate, then there is no child left in the vehicle.

[0078] In a specific implementation, the acquisition of the echo signal is specifically as follows:

[0079] The received pulses are averaged to obtain the reference received pulse, and then the target echo signal is obtained by subtracting the reference received pulse from each received pulse. The function of the received pulse is:

[0080]

[0081] Among them, i is the fast time dimension sampling point, and m is the slow time dimension time sampling point.

[0082] Furthermore, when the door signal is closed and the vehicle is started, that is, in driving state. The millimeter-wave radar and camera are jointly calibrated to achieve the spatiotemporal synchronization of radar and image data, and the target information position detected by the radar is projected into the image to determine the key area. At the same time, the frame difference method is used to detect the behavior of the child and obtain the detection area. The image of the detection area is input into the BP neural network. The neural network is sampled and trained by children's dangerous behaviors (children's heads and hands sticking out of the window, children's fighting in the car, children using knives in the car, etc.), and the detection feature map is compared and classified with the dangerous behavior feature image, that is, the dangerous behavior of children in the car is judged. When dangerous behavior occurs, the instrument pop-up window and the central control real-time panoramic display are displayed. When the dangerous behavior ends, the central control display can be turned off through the steering wheel switch.

[0083] In one or more implementations, corresponding to the above method, this embodiment provides an in-vehicle child monitoring system based on millimeter-wave radar and vision fusion, including:

[0084] A data acquisition unit, which is used to periodically acquire millimeter-wave radar data and infrared images of the area where the child is located when the vehicle is in motion;

[0085] A key focus area acquisition unit, which is used to project target information in the millimeter wave radar data into the infrared image to determine the key focus area;

[0086] A detection area acquisition unit, which is used to obtain a detection area by a frame difference method based on the focus area determined by the adjacent frame infrared images;

[0087] A dangerous behavior judgment unit, which is used to obtain a dangerous behavior judgment result by pre-training a child dangerous behavior judgment model based on a neural network based on the image of the detection area;

[0088] The alarm unit is used to issue an alarm when dangerous behavior occurs and to provide a real-time panoramic display of the children's area until the dangerous behavior ends.

[0089] It can be understood that the system described in this embodiment corresponds to the above-mentioned method embodiment, and its technical details have been described in detail in the method embodiment, so they will not be repeated here.

[0090] In further embodiments, there is also provided:

[0091] An electronic device includes a memory and a processor and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the above embodiment. For the sake of brevity, no further description is given here.

[0092] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0094] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in the above embodiment is completed.

[0095] A computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the above embodiment is implemented.

[0096] In more embodiments, a vehicle is provided, which adopts the above-mentioned in-vehicle child monitoring method based on millimeter-wave radar and vision fusion.

[0097] In a specific implementation, the vehicle may also include RF (Radio Frequency) circuits, memories including one or more computer-readable storage media, input units, display units, sensors, audio circuits, WiFi (Wireless Fidelity) modules, processors including one or more processing cores, and power supplies. Those skilled in the art will appreciate that the above components do not limit the vehicle and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. Among them:

[0098] RF circuits can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors for processing; in addition, uplink data is sent to the base station. Usually, the RF circuit includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a user identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0099] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the vehicle (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0100] The input unit can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display or touchpad, can collect the user's touch operations on or near it (such as the user's operation on or near the touch-sensitive surface using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor, and can receive and execute the command sent by the processor. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch-sensitive surface, the input unit may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0101] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the vehicle, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides corresponding visual output on the display panel according to the type of touch event. The touch-sensitive surface and the display panel are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface can be integrated with the display panel to implement input and output functions.

[0102] The vehicle may also include at least one sensor, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the vehicle moves to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the vehicle, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0103] The audio circuit, speaker, and microphone can provide an audio interface between the user and the vehicle. The audio circuit can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data. The audio data is then processed by the output processor and sent to another vehicle through the RF circuit, or the audio data is output to a memory for further processing. The audio circuit may also include an earphone jack to provide communication between an external headset and the vehicle.

[0104] WiFi is a short-range wireless transmission technology. The vehicle can help users send and receive emails, browse web pages, and access streaming media through the WiFi module, which provides users with wireless broadband Internet access. Although the WiFi module is shown, it is understandable that it is not a necessary component of the vehicle and can be omitted as needed without changing the essence of the invention.

[0105] The processor is the control center of the vehicle, using various interfaces and lines to connect various parts of the entire mobile phone, and by running or executing software programs and / or modules stored in the memory, and calling data stored in the memory, it executes various functions of the vehicle and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor may include one or more processing cores; preferably, the processor may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor.

[0106] The vehicle also includes a power source (such as a battery) for supplying power to various components. Preferably, the power source can be logically connected to the processor through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power source can also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0107] Although not shown, the vehicle may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the vehicle is a touch screen display, and the vehicle also includes a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include a method for executing the method shown in the above embodiment.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for monitoring children in a vehicle based on the fusion of millimeter-wave radar and vision, characterized in that: include: When the vehicle is in motion, the millimeter-wave radar data and infrared images of the area where the child is located are periodically acquired; Project the target information in the millimeter-wave radar data into the infrared image to determine the key areas of interest; Based on the focus area determined by the adjacent frame infrared images, the detection area is obtained by using the frame difference method; Based on the image of the detection area, the dangerous behavior judgment result is obtained by pre-training a child dangerous behavior judgment model based on a neural network; When dangerous behavior occurs, an alarm is issued and a real-time panoramic display of the children's area is provided until the dangerous behavior ends.

2. The method for monitoring children in a vehicle based on millimeter wave radar and vision fusion as claimed in claim 1, characterized in that: The method of projecting the target information in the millimeter-wave radar data into the infrared image and determining the key focus area is specifically as follows: by jointly calibrating the millimeter-wave radar and the camera, the spatiotemporal synchronization of the radar and image data is achieved, the target information position detected by the radar is projected into the image, the spatiotemporal fusion of the millimeter-wave radar data and the infrared image is achieved, and the key focus area is determined based on the fusion result.

3. The method for in-vehicle child monitoring based on millimeter-wave radar and vision fusion as claimed in claim 2, characterized in that: The spatiotemporal fusion is specifically as follows: based on the positional relationship between the millimeter radar and the camera, and the conversion relationship between the millimeter radar coordinate system and the world coordinate system, a projection matrix of the target information in the millimeter wave radar data to the infrared image is obtained; based on the obtained projection matrix, the spatial fusion of the millimeter wave radar data and the infrared image is realized; and, taking the data collected by the device with the lowest sampling frequency among the camera and the millimeter radar as a benchmark, at the collection time corresponding to the benchmark data, the collection data of another device is obtained, and the temporal fusion of the millimeter wave radar data and the infrared image is performed.

4. The method for in-vehicle child monitoring based on millimeter-wave radar and vision fusion as claimed in claim 1, characterized in that: The dangerous behavior judgment result is obtained by pre-training a child dangerous behavior judgment model based on a neural network, specifically by extracting features from an image of a detection area, using the extracted features as inputs to the child dangerous behavior judgment model, and obtaining the child dangerous behavior judgment result.

5. The method for monitoring children in a vehicle based on millimeter wave radar and vision fusion as claimed in claim 1, characterized in that: The driving state is specifically determined as follows: when the door signal is turned off and the vehicle is started, it is determined to be in the driving state; when the door signal is turned off and the vehicle is powered off, it is determined to be in the locked state.

6. The method for in-vehicle child monitoring based on millimeter-wave radar and vision fusion as claimed in claim 1, characterized in that: For the non-driving state, when the car is locked, the heart rate and breathing rate in the car are detected through the millimeter wave radar; when the breathing rate is detected to be within the preset first range and the heart rate is within the preset second frequency range, it is determined that there is a child in the car and an alarm is issued.

7. The method for in-vehicle child monitoring based on millimeter-wave radar and vision fusion as claimed in claim 6, characterized in that: The alarm adopts a cascade alarm strategy, specifically: when the car locking time is within a first preset time range, the car lights flash and the horn sounds an alarm; when the car locking time is within a second preset time range, an alarm is issued via text message or Internet phone; when the car locking time exceeds the second preset range, the car lights flash and the horn sounds an alarm, an alarm is issued via text message or Internet phone, and an alarm is issued.

8. The in-vehicle child monitoring system based on millimeter-wave radar and vision fusion is characterized by: include: A data acquisition unit, which is used to periodically acquire millimeter-wave radar data and infrared images of the area where the child is located when the vehicle is in motion; A key focus area acquisition unit, which is used to project target information in the millimeter wave radar data into the infrared image to determine the key focus area; A detection area acquisition unit, which is used to obtain a detection area by a frame difference method based on the focus area determined by the adjacent frame infrared images; A dangerous behavior judgment unit, which is used to obtain a dangerous behavior judgment result by pre-training a child dangerous behavior judgment model based on a neural network based on the image of the detection area; The alarm unit is used to issue an alarm when dangerous behavior occurs and to provide a real-time panoramic display of the children's area until the dangerous behavior ends.

9. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, the method for monitoring children in a vehicle based on the fusion of millimeter-wave radar and vision as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, an in-vehicle child monitoring method based on millimeter-wave radar and vision fusion as described in any one of claims 1 to 7 is implemented.

Images