Vehicle body moving target detection method, system and product

Abstract

The invention relates to the technical field of vehicles, artificial intelligence and vehicle operating systems, and discloses a method, a system and a product for detecting moving targets around a vehicle. The method comprises the following steps: detecting a moving target in an original image, judging the type of the moving target, and generating a to-be-detected image based on the moving target; judging whether the moving target is a normal pedestrian or an abnormal moving user; predicting the moving direction of the abnormal moving user; obtaining the current distance and speed between the abnormal mobile user and the target vehicle; according to the moving direction, the current speed and the distance of the abnormal moving user, whether potential danger exists or not is judged, and early warning information is generated; and according to the early warning information, prompting through at least two of the following interaction modes: playing prompt voice, displaying prompt information, and controlling a steering wheel to vibrate for prompting. According to the system, early warning of potential risks possibly occurring in the vehicle driving process is achieved, and the accident risk is reduced.

Description

Technical Field

[0001] The present application relates to the fields of vehicles, artificial intelligence technology, and vehicle operating system technology, and in particular to a method, system, and product for detecting moving targets around a vehicle. Background Art

[0002] In recent years, the use of personal mobility devices (such as electric vehicles and bicycles) has increased. There are certain safety risks when personal mobility devices and cars travel on the road together. Since personal mobility devices are usually more difficult to predict and identify than ordinary pedestrians, they bring more stringent safety challenges to car drivers.

[0003] Since personal mobility devices move faster than pedestrians, drivers pay attention to multiple information sources while driving, which causes distraction and makes it difficult to accurately judge and quickly respond to the fast-moving personal mobility devices around the vehicle, resulting in an increase in the risk of accidents. Therefore, it is necessary to find a way to improve the driver's ability to recognize and respond to the surrounding environment while the car is driving, improve road safety, and reduce the probability of accidents. Summary of the invention

[0004] In view of this, the present application aims to propose a method, system and product for detecting moving targets around a vehicle, so as to improve the driver's cognition and reaction ability to his surrounding environment, improve road safety and reduce the probability of accidents.

[0005] To achieve the above purpose, the technical solution of this application is as follows:

[0006] A first aspect of an embodiment of the present application provides a method for detecting a moving target around a vehicle, the method comprising:

[0007] Detecting a moving target in an original image, determining the type of the moving target, and generating an image to be detected based on the moving target; the original image is an image of the surrounding environment collected in real time during the movement of the target vehicle;

[0008] Processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal moving user;

[0009] Detecting the image to be detected of the abnormal mobile user and predicting the moving direction;

[0010] Obtaining the current distance and speed between the abnormal mobile user and the target vehicle; judging whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; generating warning information if there is a potential danger;

[0011] According to the warning information, prompts are given through at least two of the following interactive methods: playing prompt voice, displaying prompt information, and controlling the steering wheel to vibrate for prompt.

[0012] Optionally, processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal moving user includes:

[0013] Reconstructing corresponding expected pedestrian data according to the image to be detected by using a first detection model;

[0014] Calculating a difference score between the expected pedestrian data and the image to be detected;

[0015] Compare the difference score with the difference threshold to obtain a first detection result; the first detection result is: a normal pedestrian or an abnormal mobile user;

[0016] When the first detection result is a normal pedestrian, the parameters of the first detection model are updated based on the image to be detected.

[0017] Optionally, generating an image to be detected based on each moving target includes:

[0018] Obtaining the current distance between the moving target and the target vehicle;

[0019] Compare the distance with a distance threshold, and filter out moving targets whose distance is not greater than the distance threshold;

[0020] Generate corresponding images to be detected based on each selected moving target.

[0021] Optionally, detecting the image to be detected of the abnormal mobile user and predicting the moving direction includes:

[0022] Using a second detection model, identifying the human body posture in the image to be detected of the abnormal mobile user, and extracting intermediate features;

[0023] According to the intermediate features, the scores of the moving direction of the abnormal mobile user in each candidate direction are calculated to obtain a second prediction result; the second prediction result is used to indicate the moving direction of the abnormal mobile user.

[0024] Optionally, judging whether there is a potential danger according to the moving direction, current speed and distance of the abnormally moving user includes:

[0025] According to the moving direction of the abnormal moving user and the moving direction of the target vehicle, determining whether there is a direction intersection;

[0026] In the case of a direction intersection, the approach time is calculated according to the current distance and speed between the abnormal mobile user and the target vehicle; the approach time is compared with a safety threshold to determine whether there is a potential danger.

[0027] Optionally, comparing the approach time with a safety threshold to determine whether there is a potential danger includes:

[0028] Determining a corresponding security threshold according to the type of the abnormal mobile user;

[0029] The approach time is compared with the safety threshold, and when the approach time is not greater than the safety threshold, it is determined that there is a potential danger.

[0030] Optionally, generate warning information in the event of potential danger, including:

[0031] Calculate the danger score according to the type of abnormal mobile users with potential danger and the corresponding approach time;

[0032] Sort each abnormal mobile user according to the risk score from high to low;

[0033] Generate warning information based on the information of abnormal mobile users with the highest risk scores.

[0034] According to a second aspect of an embodiment of the present application, a vehicle surrounding moving target detection system is provided, which is used to implement the method provided in the first aspect of the embodiment of the present application, and the system includes:

[0035] A processing module, used to detect a moving target in an original image, determine the type of the moving target, and generate an image to be detected based on the moving target; the original image is an image of the surrounding environment collected in real time during the movement of the target vehicle;

[0036] An abnormality detection module, used to process the image to be detected of each moving target, and determine whether the moving target is a normal pedestrian or an abnormal moving user;

[0037] A direction prediction module, used to detect the image to be detected of the abnormal mobile user and predict the moving direction;

[0038] An analysis module is used to obtain the current distance and speed between the abnormal mobile user and the target vehicle; determine whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; and generate warning information when there is a potential danger;

[0039] The prompt module is used to give a prompt according to the warning information through at least two of the following interactive methods: playing a prompt voice, displaying prompt information, and controlling the steering wheel to vibrate to give a prompt.

[0040] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the first aspect of the embodiment of the present application are implemented.

[0041] According to the fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the method provided in the second aspect of the embodiment of the present application are implemented.

[0042] The vehicle surrounding moving target detection method provided by the present application detects the surrounding environment images (original images) collected during the driving process of the target vehicle, identifies the moving targets therein, and generates an image to be detected based on each moving target. Anomaly detection is performed on each image to be detected to determine whether the moving target is a normal pedestrian or an abnormal mobile user (such as an electric vehicle, a bicycle, etc.). For abnormal mobile users, the moving direction is further predicted, and a comprehensive analysis is performed based on the moving direction of the abnormal mobile user, the current distance from the target vehicle, and the speed to determine whether there is a potential danger. If it is determined that there is a potential danger, a warning message is generated, and the car driver is prompted through various interactive methods such as voice, display screen, and steering wheel vibration.

[0043] This system processes the original images collected in real time during the driving process of the car to determine whether there is a potential collision risk between the target vehicle and the surrounding abnormal moving users in the current state, enhance the driver's perception of the driving environment, reduce the driver's attention burden, and enable the driver to focus more on the road conditions. In addition, a variety of interactive methods are used to remind the driver during the warning, and the transmission efficiency and accuracy of information are improved through multi-sensory participation, which strengthens the driver's attention to potential dangers so that they can react in advance to avoid accidents and improve road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 is a flow chart of a method for detecting moving targets around a vehicle proposed in one embodiment of the present application;

[0046] Figure 2 is a flowchart of processing an original image in one embodiment of the present application;

[0047] Figure 3 is a schematic diagram of segmenting an image to be detected in one embodiment of the present application;

[0048] Figure 4 is a schematic diagram of a vehicle surrounding moving target detection system proposed in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0052] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects as detailed in the present application.

[0054] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0055] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0056] Figure 1 FIG. 1 is a flow chart of a method for detecting moving targets around a vehicle proposed in an embodiment of the present application. Figure 1 As shown, the method includes:

[0057] S1: Detecting a moving target in an original image, determining the type of the moving target, and generating an image to be detected based on the moving target; the original image is an image of the surrounding environment collected in real time during the movement of the target vehicle;

[0058] S2: Processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal moving user;

[0059] S3: Detect the image of the abnormal mobile user to be detected and predict the moving direction;

[0060] S4: obtaining the current distance and speed between the abnormal mobile user and the target vehicle; judging whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; generating warning information if there is a potential danger;

[0061] S5: Prompt according to the warning information through at least two of the following interactive methods: play prompt voice, display prompt information, and control the steering wheel to vibrate to give a prompt.

[0062] In this embodiment, an image of the surrounding environment of the target vehicle is obtained by a vehicle-mounted camera or radar as an original image. The moving target in the original image is detected by deep learning technology, the type of the moving target is determined and a corresponding image to be detected is generated. In this embodiment, the moving target includes: moving pedestrians, moving bicycles / electric vehicles, etc.

[0063] Figure 2 This is a flowchart for processing the original image in one embodiment of the present application. In this embodiment, the Mask R-CNN model is used as the detection model to perform human body detection on the original image and identify the relevant areas belonging to the human body. The Mask R-CNN model has two branches: a classification branch and an extraction branch. Among them, the classification branch is responsible for identifying and classifying moving targets and determining the type of moving targets. For example, the type of moving target is identified as "pedestrian", "moving bicycle" or "moving electric vehicle". The extraction branch is used to generate accurate boundary masks for each moving target in the original image to extract the image to be detected.

[0064] The Mask R-CNN model uses CNN (Convolutional Neural Networks) as the backbone network for feature extraction to extract features from the original image. The region of interest is extracted for the mobile target through RPN (Region Proposal Network), and the feature map is input into the pooling layer for region pooling to ensure that the spatial information corresponding to the feature is not lost, and a fixed-size feature map is obtained. Finally, the fixed-size feature map is input into the classification branch and the extraction branch for processing, and the type of the mobile target is obtained through the classification branch, and the boundary mask based on the outline of the mobile target is generated through the extraction branch. Furthermore, the background pixels are removed based on the boundary mask, and the pixel area related to the mobile target (such as people and bicycles) is retained to obtain the image to be detected.

[0065] Furthermore, anomaly detection is performed on the image to be detected to determine whether the moving target is a normal pedestrian or an abnormal mobile user. This solution aims to analyze the potential dangers of mobile users (such as cyclists, people riding electric bikes, etc.) around the vehicle whose moving speed is faster than that of normal pedestrians. Therefore, other mobile users who are not pedestrians are judged as abnormal mobile users, and the next moving direction of each abnormal mobile user is further predicted. Finally, based on the moving direction of the abnormal user, the current driving speed, and the distance from the target vehicle, a comprehensive analysis is performed to determine whether there is a potential danger, that is, whether an early warning is required. Early warning information is generated, and a variety of interactive methods are used to prompt the driver of the target vehicle to ensure that the driver pays attention to potential dangers in time and responds in time.

[0066] This application provides three different interaction methods, including: playing prompt voice in the target vehicle, using AR-HUD technology to project prompt information on the windshield of the target vehicle, and controlling the steering wheel to vibrate. Compared with the single reminder method of only playing prompt audio through the vehicle voice module in the related scheme, this application provides multiple interaction modes for early warning. By combining multiple interaction methods, drivers can notice potential dangers more quickly, which facilitates drivers to respond quickly and reduce the accident rate.

[0067] Traditional target detection and semantic segmentation solutions require the acquisition of video data or multiple frames of continuous images to predict dynamic information. Their detection efficiency cannot meet the real-time requirements of vehicle driving, and most traditional detection solutions can only identify the type of static objects and cannot detect moving targets. This embodiment uses a deep learning model to perform abnormal detection and moving direction detection of moving targets, which can more accurately identify abnormal mobile users such as bicycle riders and electric vehicle riders, and predict their next moving direction. It realizes the prediction of dynamic information based on a single image of a moving target in motion, which can well meet the real-time requirements of the vehicle driving process.

[0068] Optionally, when collecting the original image, a laser radar enhancement system (Light Detection And Ranging, LIDAR) can also be used. This radar system can obtain high-precision three-dimensional data of the vehicle's surrounding environment in real time. Then, combined with a deep learning model to process the high-precision three-dimensional data of the vehicle's surrounding environment, it can more accurately identify and predict the movement of individual abnormal mobile users.

[0069] As an implementation manner of the present application, processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal mobile user includes:

[0070] Reconstructing corresponding expected pedestrian data according to the image to be detected by using a first detection model;

[0071] Calculating a difference score between the expected pedestrian data and the image to be detected;

[0072] Compare the difference score with the difference threshold to obtain a first detection result; the first detection result is: a normal pedestrian or an abnormal mobile user;

[0073] When the first detection result is a normal pedestrian, the parameters of the first detection model are updated based on the image to be detected.

[0074] In one embodiment, deep learning technology is used to perform abnormality detection on the image to be detected to determine whether the moving target is a normal pedestrian or an abnormal mobile user. An abnormal mobile user refers to a mobile user with a significantly different behavior pattern from a typical pedestrian, such as a cyclist, a person riding an electric bike, etc.

[0075] Specifically, the DMNAD (Dynamic Memory-guided Normality for Anomaly Detection) model is used as the first detection model. During training, the model only trains normal pedestrian patterns, but not abnormal mobile user patterns. This training method can effectively distinguish pedestrians and individual mobile users in a dynamic environment.

[0076] For the input image, the DMNAD model reconstructs a normal pedestrian output (expected pedestrian data) based on the neural network, and then calculates the difference score between the input image and the expected pedestrian data. The difference score is compared with the difference threshold. If the difference score is greater than the difference threshold, it is determined to be an abnormal mobile user; if the difference score is not greater than the difference threshold, it is determined to be a normal pedestrian. In addition, the DMNAD model has a dynamic memory module for continuously updating and improving the learning of the "normal" pedestrian pattern. In the case where the moving target is a normal pedestrian, the model updates the parameters of the dynamic memory module inside the model based on the information of the input image, thereby continuously improving the detection performance of the model during the detection process.

[0077] Compared with traditional rule-based detection and machine learning methods, the traditional method of "manually setting behavioral features and training models based on rule-based methods" is difficult to adapt to the complex environment around the car driving process. In this embodiment, deep learning technology is used to adaptively capture dynamic behavior patterns through the DMNAD model. The model only needs to learn the regular "normal pedestrian" pattern, and there is no need to train other irregular "abnormal mobile user" patterns that are different from "normal pedestrians" one by one. It can adapt well to abnormal behaviors in complex backgrounds and environments, especially suitable for the dynamic environment of vehicle driving. It can quickly adapt to a variety of abnormal moving targets, reduce the complexity of the model, and meet the needs of real-time detection.

[0078] In one embodiment, after the image to be detected with the background removed is extracted by the processing module, the image to be detected is further segmented to obtain multiple sub-images of local regions. Specifically, each image to be detected is segmented into two sub-images: an upper sub-image and a lower sub-image. Further, the lower sub-image is input into the first detection model for abnormality detection.

[0079] Figure 3 FIG. 1 is a schematic diagram of segmenting an image to be detected in one embodiment of the present application. Figure 3 As shown in FIG, by locating the human body in the image to be detected, the upper and lower body parts of the human body are segmented to obtain the upper sub-image and the lower sub-image. The upper sub-image retains the posture information of the human body, and the lower sub-image retains the information of the moving tool (such as Figure 3In the subsequent abnormal detection, in order to determine whether the moving target is a normal pedestrian or an abnormal moving user, it is necessary to capture the information of the local moving tool (such as a bicycle). Therefore, only the lower sub-image obtained by segmentation needs to be input into the first detection model for detection. By segmenting the image to be detected and using the sub-image of the local area for detection, the calculation amount of the first detection model is reduced and the detection efficiency is improved.

[0080] In one embodiment, the upper subgraph is linked to the lower subgraph. Figure 1 Since the upper sub-image retains the posture information of the human body, the posture information of the human body in the upper sub-image is captured during detection to supplement the information extracted from the lower sub-image, so as to improve the accuracy of abnormality detection.

[0081] As an implementation of the present application, generating an image to be detected based on each moving target includes:

[0082] Obtaining the current distance between the moving target and the target vehicle;

[0083] Compare the distance with a distance threshold, and filter out moving targets whose distance is not greater than the distance threshold;

[0084] Generate corresponding images to be detected based on each selected moving target.

[0085] In one embodiment, there are moving targets that are far away from the target vehicle in the collected original image. Since the probability of such moving targets that are far away colliding with the target vehicle is very small, such moving targets can be eliminated in the initial processing stage, eliminating the subsequent abnormality detection and direction prediction operations for such moving targets.

[0086] Specifically, while collecting the original image, the distance between each moving target and the target vehicle is obtained through the vehicle-mounted radar, and each moving target is screened by the distance threshold. Only the region of interest is extracted for the moving targets within the distance threshold range, and the corresponding image to be detected is generated. By removing moving targets that are far away in the initial stage of generating the image to be detected, the calculation amount of each subsequent detection model is reduced, thereby improving the detection efficiency.

[0087] As an implementation manner of the present application, detecting the image to be detected of the abnormal mobile user and predicting the moving direction includes:

[0088] Using a second detection model, identifying the human body posture in the image to be detected of the abnormal mobile user, and extracting intermediate features;

[0089] According to the intermediate features, the scores of the moving direction of the abnormal mobile user in each candidate direction are calculated to obtain a second prediction result; the second prediction result is used to indicate the moving direction of the abnormal mobile user.

[0090] In one embodiment, deep learning technology is used to process the image to be detected to predict the next moving direction of the abnormal moving user. Specifically, the MEBOW (Monocular Estimation of Body Orientation) model is used as the second detection model. The model includes: HRNet (High-Resolution Network) and ResNet (Residual Network), wherein the HRNet network is the backbone network, which is used to identify the posture of a single individual (moving target) from the image to be detected and extract intermediate features. Then, the intermediate features extracted by HRNet are fused into the ResNet head network, and the extracted intermediate features are processed by the ResNet network to predict the next moving direction of the abnormal moving target. The ResNet network solves the problems of gradient vanishing and gradient explosion in the deep network training process by introducing residual connections, effectively deepening the depth of the network and improving the accuracy of direction prediction.

[0091] In practical applications, the number of candidate items of the moving direction can be set as needed, and this application does not impose any limitation on this. Optionally, in this embodiment, the candidate items of the output result are set to 4-8 horizontal directions.

[0092] Compared with the traditional rule-based method for direction prediction, the traditional rule-based method usually requires the pre-definition of multiple behavioral features (for example, speed, acceleration, etc.), and it is often difficult to cover all situations in complex environments. For example, it performs poorly when dealing with the complexity of human posture and performs poorly in the face of changing environments (such as shadows, occlusions, etc.). However, this embodiment uses the MEBOW model based on deep learning to automatically learn the characteristics of human posture, without relying on manually set rules. The model uses the posture estimation of key points to more accurately judge the direction of the human body, and can better predict the movement trend of moving targets, and can obtain accurate detection results even under different background and lighting conditions.

[0093] As an implementation mode of the present application, judging whether there is a potential danger according to the moving direction, current speed and distance of the abnormal moving user includes:

[0094] According to the moving direction of the abnormal moving user and the moving direction of the target vehicle, determining whether there is a direction intersection;

[0095] In the case of a direction intersection, the approach time is calculated according to the current distance and speed between the abnormal mobile user and the target vehicle; the approach time is compared with a safety threshold to determine whether there is a potential danger.

[0096] In one embodiment, after the moving direction of the abnormal mobile user is predicted by the second detection model, it is determined whether there is a potential danger by combining the speed of the abnormal mobile user and the distance from the target vehicle. The detection results of each detection model and the data collected by the sensor (such as the vehicle-mounted radar) are comprehensively analyzed to determine whether an early warning is needed.

[0097] Specifically, the moving direction of the target vehicle is obtained, and combined with the moving direction of the abnormal user, it is determined whether there is an intersection (i.e., a collision may occur). If there is a direction intersection, the approach time is calculated based on the moving speed of the current abnormal user obtained by the radar system and the distance from the target vehicle, combined with the driving speed of the target vehicle. Further, the approach time is compared with the safety threshold. If the approach time is less than or equal to the safety threshold, it is determined that there is a potential danger and an early warning is required.

[0098] In one embodiment, the vehicle may accelerate or decelerate during driving. Therefore, in addition to detecting the speed and distance of the abnormally moving user, the acceleration of the abnormally moving user can be calculated by continuously capturing multiple frames of images, and then combined with the acceleration to calculate a more accurate approach time, thereby further improving vehicle safety.

[0099] As an implementation manner of the present application, comparing the approach time with a safety threshold to determine whether there is a potential danger includes:

[0100] Determining a corresponding security threshold according to the type of the abnormal mobile user;

[0101] The approach time is compared with the safety threshold, and when the approach time is not greater than the safety threshold, it is determined that there is a potential danger.

[0102] In one embodiment, a corresponding safety threshold is determined according to the type of abnormal mobile user. Since different types of mobile users have different motion characteristics during driving, for example, a motorcycle may suddenly accelerate or decelerate during driving, while a bicycle usually does not suddenly change speed during movement. Based on this, in this embodiment, corresponding safety thresholds are set according to different types of mobile targets. After the approach time is calculated, the corresponding safety threshold is determined in combination with the type of the abnormal mobile user, and the approach time is compared with the safety threshold. When the approach time is less than or equal to the safety threshold, it is determined that there is a potential danger.

[0103] By setting different safety thresholds according to the type of moving target, drivers can respond to different types of emergencies in a timely manner, further improving road safety.

[0104] As an implementation of the present application, generating warning information in the event of a potential danger includes:

[0105] Calculate the danger score according to the type of abnormal mobile users with potential danger and the corresponding approach time;

[0106] Sort each abnormal mobile user according to the risk score from high to low;

[0107] Generate warning information based on the information of abnormal mobile users with the highest risk scores.

[0108] In one embodiment, when there are multiple potentially dangerous abnormal mobile users, in order to avoid distraction due to excessive information that the driver is paying attention to, an abnormal mobile user with the highest degree of danger is obtained through screening and an early warning is issued.

[0109] The corresponding scoring intervals are set in advance based on the length of approach time and the different types of abnormal mobile users. For example, the score of electric vehicles is higher than that of bicycles. The danger score is calculated based on the approach time and type of abnormal mobile users, and they are sorted from high to low according to the danger score, and the abnormal mobile users with the highest danger score are selected as the mobile targets that need to be warned at present.

[0110] By calculating the danger score and screening the abnormal mobile users with the highest current danger level for early warning, the driver can be helped to prioritize early warnings for the abnormal mobile users with the highest danger level when there is potential danger without being disturbed by too much information, so that the driver can quickly capture the most important information and make correct decisions at critical moments.

[0111] In this embodiment, early warning prompts are provided through a variety of interactive methods, including voice interaction, AR-HUD display information interaction, and slight vibration of the steering wheel.

[0112] Voice interaction is to generate prompt audio according to the warning information through a deep learning speech synthesis model. Compared with visual prompts, playing voice can prevent the driver from shifting too much attention from the road to the display screen. Optionally, the voice module also integrates voice interaction function, which supports the driver to confirm the receipt of prompts and query more information through voice commands, further enhancing the driver's sense of control and improving emergency response efficiency.

[0113] AR-HUD interaction is to display the prompt information in the driver's field of vision based on the warning information. Considering that the windshield is within the driver's field of vision, the prompt information is projected at a fixed position on the windshield for display, and the prompt information is simplified to only display the most critical information to avoid excessive information from interfering with the driver, enhance the driver's perception and prediction of road conditions, and improve driving safety. Compared with voice prompts, it can attract the driver's attention more quickly.

[0114] Steering wheel vibration interaction mainly controls the steering wheel to vibrate slightly according to the warning information to attract the driver's attention. The frequency and intensity of the vibration are dynamically adjusted according to the danger score, and the driver's attention is attracted through vibration feedback, while ensuring that normal steering wheel operation is not interfered with. Compared with voice and display prompts, steering wheel vibration prompts are more concealed and will not occupy the driver's sight and attention too much. This prompt method can be combined with other prompt methods as an auxiliary interactive means to provide drivers with more comprehensive and intelligent danger warnings.

[0115] In one embodiment, the generated warning information is not only used for warning prompts inside the target vehicle, but also processed to generate driving data and sent to fixed receiving devices or mobile handheld devices in the vicinity of the target vehicle, and communicates with the traffic infrastructure around the target vehicle, other road users or mobile handheld devices. By communicating with the traffic infrastructure, the road management system can uniformly control the road driving conditions and reduce the accident rate.

[0116] For example, the traffic management system obtains the driving data of multiple vehicles on the road, analyzes and determines if there are safety hazards, and then controls the traffic lights or speed limit thresholds in some areas to improve road safety.

[0117] Other road users or mobile handheld devices receive driving data in the vicinity, and can be informed of surrounding traffic conditions in a timely manner and predict possible dangers in advance. For example, information interaction with the navigation software in the mobile handheld device allows pedestrians who are using the navigation software to receive danger warnings and avoid them in time.

[0118] Based on the same inventive concept, an embodiment of the present application provides a system for detecting moving objects around a vehicle. Figure 4 FIG. 1 is a schematic diagram of a vehicle surrounding moving target detection system 100 proposed in an embodiment of the present application. Figure 4 As shown, the system includes:

[0119] The processing module 101 is used to detect a moving target in an original image, determine the type of the moving target, and generate an image to be detected based on the moving target; the original image is an image of the surrounding environment collected in real time during the movement of the target vehicle;

[0120] The abnormality detection module 102 is used to process the image to be detected of each moving target and determine whether the moving target is a normal pedestrian or an abnormal moving user;

[0121] A direction prediction module 103, used to detect the image to be detected of the abnormal mobile user and predict the moving direction;

[0122] The analysis module 104 is used to obtain the current distance and speed between the abnormal mobile user and the target vehicle; determine whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; and generate warning information if there is a potential danger;

[0123] The prompt module 105 is used to give a prompt according to the warning information through at least two of the following interactive modes: playing a prompt voice, displaying a prompt message, and controlling the steering wheel to vibrate to give a prompt.

[0124] As an implementation manner of the present application, the anomaly detection module 102 includes a first detection model pre-trained based on a neural network; the first detection model is specifically used to perform the following steps:

[0125] reconstructing corresponding expected pedestrian data according to the image to be detected;

[0126] Calculating a difference score between the expected pedestrian data and the image to be detected;

[0127] Compare the difference score with the difference threshold to obtain a first detection result; the first detection result is: a normal pedestrian or an abnormal mobile user;

[0128] When the first detection result is a normal pedestrian, the parameters of the first detection model are updated based on the image to be detected.

[0129] As an implementation manner of the present application, the processing module 101 further includes a segmentation submodule, which is used to segment the image to be detected into upper and lower parts to obtain an upper sub-image and a lower sub-image;

[0130] The anomaly detection module 102 is specifically configured to process the lower sub-image of the image to be detected.

[0131] As an implementation manner of the present application, the processing module 101 is used to generate an image to be detected based on each moving target, specifically including:

[0132] Obtaining the current distance between the moving target and the target vehicle;

[0133] Compare the distance with a distance threshold, and filter out moving targets whose distance is not greater than the distance threshold;

[0134] Generate corresponding images to be detected based on each selected moving target.

[0135] As an implementation manner of the present application, the direction prediction module 103 includes a second detection model pre-trained based on a neural network; the second detection model is specifically used to perform the following steps:

[0136] Identifying the human body posture in the image to be detected of the abnormal mobile user and extracting intermediate features;

[0137] According to the intermediate features, the scores of the moving direction of the abnormal mobile user in each candidate direction are calculated to obtain a second prediction result; the second prediction result is used to indicate the moving direction of the abnormal mobile user.

[0138] As an implementation manner of the present application, the analysis module 104 is used to determine whether there is a potential danger according to the moving direction, current speed and distance of the abnormal moving user, specifically including:

[0139] According to the moving direction of the abnormal moving user and the moving direction of the target vehicle, determining whether there is a direction intersection;

[0140] In the case of a direction intersection, the approach time is calculated according to the current distance and speed between the abnormal mobile user and the target vehicle; the approach time is compared with a safety threshold to determine whether there is a potential danger.

[0141] As an implementation manner of the present application, the analysis module 104 is used to compare the approach time with the safety threshold to determine whether there is a potential danger, specifically including:

[0142] Determining a corresponding security threshold according to the type of the abnormal mobile user;

[0143] The approach time is compared with the safety threshold, and when the approach time is not greater than the safety threshold, it is determined that there is a potential danger.

[0144] As an implementation of the present application, the analysis module 104 is used to generate warning information in the event of a potential danger, specifically including:

[0145] Calculate the danger score according to the type of abnormal mobile users with potential danger and the corresponding approach time;

[0146] Sort each abnormal mobile user according to the risk score from high to low;

[0147] Generate warning information based on the information of abnormal mobile users with the highest risk scores.

[0148] As an implementation mode of the present application, the prompt module 105 includes at least two of the following: a voice module, a display module or a vibration module;

[0149] The voice module is used to play a prompt voice in the target vehicle according to the warning information;

[0150] The display module is used to display prompt information at a specified position in the target vehicle using AR-HUD technology according to the warning information;

[0151] The vibration module is used to determine the vibration frequency and vibration intensity according to the early warning information; and control the steering wheel vibration prompt of the target vehicle according to the vibration frequency and vibration intensity.

[0152] In one embodiment, the vehicle surrounding moving target detection system also includes a communication module for generating driving data based on the warning information and sending it to a target device; the target device includes at least one of the following: a fixed receiving device or a mobile handheld device within a specified distance range of the target vehicle.

[0153] Based on the same inventive concept, an embodiment of the present application provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the steps in the vehicle surrounding moving target detection method as described in any of the above embodiments of the present application are implemented.

[0154] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting moving targets around a vehicle as described in any of the above embodiments of the present application are implemented.

[0155] Based on the same inventive concept, an embodiment of the present application provides an electronic device, referring to Figure 5 , Figure 5 Schematic diagram of an electronic device proposed in one embodiment of the present application. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the steps in the method for detecting a moving target around a vehicle as described in any of the above embodiments of the present application are implemented.

[0156] Regarding the method in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the system, and will not be elaborated here.

[0157] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0158] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily required by the present application.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0160] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0163] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the present application is interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0165] The above is a detailed introduction to the vehicle surrounding moving target detection method, system and product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting moving targets around a vehicle, characterized in that: include: Detecting a moving target in an original image, determining the type of the moving target, and generating an image to be detected based on the moving target; The original image is an image of the surrounding environment collected in real time during the movement of the target vehicle; Processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal moving user; Detecting the image to be detected of the abnormal mobile user and predicting the moving direction; Obtaining the current distance and speed between the abnormal mobile user and the target vehicle; judging whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; generating warning information if there is a potential danger; According to the warning information, prompts are given through at least two of the following interactive methods: playing prompt voice, displaying prompt information, and controlling the steering wheel to vibrate for prompt.

2. The method for detecting moving targets around a vehicle according to claim 1, characterized in that: Processing the image to be detected of each moving target to determine whether the moving target is a normal pedestrian or an abnormal moving user includes: Reconstructing corresponding expected pedestrian data according to the image to be detected by using a first detection model; Calculating a difference score between the expected pedestrian data and the image to be detected; Compare the difference score with the difference threshold to obtain a first detection result; the first detection result is: a normal pedestrian or an abnormal mobile user; When the first detection result is a normal pedestrian, the parameters of the first detection model are updated based on the image to be detected.

3. The method for detecting moving targets around a vehicle according to claim 1, characterized in that: Generate an image to be detected based on each moving target, including: Obtaining the current distance between the moving target and the target vehicle; Compare the distance with a distance threshold, and filter out moving targets whose distance is not greater than the distance threshold; Generate corresponding images to be detected based on each selected moving target.

4. The method for detecting moving targets around a vehicle according to claim 1, characterized in that: Detecting the image to be detected of the abnormal mobile user and predicting the moving direction include: Using a second detection model, identifying the human body posture in the image to be detected of the abnormal mobile user, and extracting intermediate features; According to the intermediate features, the scores of the moving direction of the abnormal mobile user in each candidate direction are calculated to obtain a second prediction result; the second prediction result is used to indicate the moving direction of the abnormal mobile user.

5. The method for detecting moving targets around a vehicle according to claim 1, characterized in that: According to the moving direction, current speed and distance of the abnormal moving user, determine whether there is a potential danger, including: According to the moving direction of the abnormal moving user and the moving direction of the target vehicle, determining whether there is a direction intersection; In the case of a direction intersection, the approach time is calculated according to the current distance and speed between the abnormal mobile user and the target vehicle; the approach time is compared with a safety threshold to determine whether there is a potential danger.

6. The method for detecting moving targets around a vehicle according to claim 5, characterized in that: Compare the approach time with a safety threshold to determine whether there is a potential hazard, including: Determining a corresponding security threshold according to the type of the abnormal mobile user; The approach time is compared with the safety threshold, and when the approach time is not greater than the safety threshold, it is determined that there is a potential danger.

7. The method for detecting moving targets around a vehicle according to claim 6, characterized in that: Generates early warning messages in potentially hazardous situations, including: Calculate the danger score according to the type of abnormal mobile users with potential danger and the corresponding approach time; Sort each abnormal mobile user according to the risk score from high to low; Generate warning information based on the information of abnormal mobile users with the highest risk scores.

8. A vehicle-surrounding moving target detection system, characterized in that: Used to implement the method according to any one of claims 1 to 7, comprising: A processing module, used to detect a moving target in an original image, determine the type of the moving target, and generate an image to be detected based on the moving target; the original image is an image of the surrounding environment collected in real time during the movement of the target vehicle; An abnormality detection module, used to process the image to be detected of each moving target, and determine whether the moving target is a normal pedestrian or an abnormal moving user; A direction prediction module, used to detect the image to be detected of the abnormal mobile user and predict the moving direction; An analysis module is used to obtain the current distance and speed between the abnormal mobile user and the target vehicle; determine whether there is a potential danger according to the moving direction, current speed and distance of the abnormal mobile user; and generate warning information when there is a potential danger; The prompt module is used to give a prompt according to the warning information through at least two of the following interactive methods: playing a prompt voice, displaying prompt information, and controlling the steering wheel to vibrate to give a prompt.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps in the method according to any one of claims 1 to 7 are implemented.

Images