Vehicle-based environment sensing method, device, equipment and medium

Abstract

The invention relates to the field of environment perception, in particular to an environment perception method, device and equipment based on a vehicle and a medium. According to the invention, through cooperative work of the radar device and the camera device, multi-dimensional feature collection of the candidate target is realized, the accuracy of target analysis is improved, and the problems of false alarm and false recording can be effectively reduced. And secondly, the radar equipment is used for detecting the candidate target and the first object feature, and the camera equipment is used for acquiring the second object feature, so that the function of detecting the moving distance, speed and category of the object can be realized only aiming at UWB radar adjustment without a complex algorithm. And then, through the combined action of the radar equipment and the camera equipment, the defect of insufficient detection distance of the UWB radar is made up, the detection range of the surrounding environment is expanded, and the risk of abnormal target intrusion can be effectively alerted. In addition, the radar and the camera device collect features of the front end and the rear end of the target at different stages, and whole-course tight coupling recognition monitoring of the suspicious object from the far end to the near end is achieved.

Description

Technical Field

[0001] The present invention relates to the field of environmental perception, and in particular to a vehicle-based environmental perception method, device, equipment and medium. Background Art

[0002] In the process of the development of vehicle intelligence, the UWB radar and key system have been gradually applied. This system accurately determines the position of the digital key by measuring the transmission time of sending and receiving short pulses to realize functions such as vehicle unlocking and starting. Its radar detection range is generally between 0.1m and 10m, which can basically meet the detection requirements of the surrounding environment for the conventional ordinary urban parking sentry system.

[0003] However, this system has many limitations. On the one hand, the function of the UWB radar in the digital key system is often limited to only detecting and positioning the digital key. If you want to realize functions such as detecting the moving distance, speed and category of other surrounding objects, you not only need to adjust the radar working mode, but also need to develop additional detection algorithms, and even be equipped with relevant hardware and software support, and the algorithm for distinguishing object categories is extremely complex. On the other hand, in the sentry scenario of outdoor camping camps, the detection distance of the UWB digital key radar is too small to meet the detection requirements for a wider surrounding environment and cannot effectively guard against the risk of large wild animals invading. In addition, the UWB digital key radar cannot perform tight-coupling identification and monitoring of suspicious objects from the far end to the near end of the vehicle, and there is no multi-sensor perception fusion verification at the near end of the vehicle, which is very likely to cause false alarms and false records. Summary of the Invention

[0004] In view of this, the embodiments of the present invention provide a vehicle-based environmental perception method, device, equipment and medium to solve the problems that the function expansion of the UWB radar in the digital key system is limited, the algorithm is complex, the detection distance is insufficient in the sentry scenario of outdoor camping camps, and it cannot perform full-process tight-coupling identification and monitoring of suspicious objects, and there is a lack of multi-sensor perception fusion verification at the near end, which is prone to false alarms and false records.

[0005] In a first aspect, the embodiments of the present invention provide a vehicle-based environmental perception method, and the method includes:

[0006] When the vehicle enters the sentry mode, call the radar device deployed on the vehicle to detect at least one candidate target in the environment where the vehicle is located and the first object feature of the candidate target;

[0007] Obtain the target warning area where the candidate target is currently located, and determine whether the detection condition is satisfied at the current moment based on the target warning area;

[0008] If the current moment meets the detection condition, control the camera device deployed on the vehicle to detect the candidate target, and obtain the second object feature of the candidate target;

[0009] Analyze the candidate target based on the first object feature and the second object feature to obtain an analysis result, and perform corresponding safety warning operations based on the analysis result, where the analysis result is used to indicate whether the candidate target is an abnormal target.

[0010] Further, determining whether the current moment meets the detection condition based on the target warning area includes:

[0011] Detect whether the target warning area is a first warning area;

[0012] If the target warning area is a first warning area, determine that the current moment does not meet the detection condition;

[0013] If the target warning area is a second warning area, determine that the current moment meets the detection condition; or, if the target warning area is a third warning area, determine that the current moment meets the detection condition;

[0014] Wherein, the first warning area, the second warning area, and the third warning area are warning areas constructed with the vehicle as the center and different distances as the radii; the first warning area includes the second warning area, and the second warning area includes the third warning area.

[0015] Further, controlling the camera device deployed on the vehicle to detect the candidate target and obtaining the second object feature of the candidate target includes:

[0016] If the target warning area is a second warning area, obtain the first object identifier when the candidate target enters the first warning area and the second object identifier when the candidate target is in the second warning area;

[0017] Compare the first object identifier and the second object identifier;

[0018] If the first object identifier is the same as the second object identifier, send a control instruction to the camera device deployed on the vehicle, so that the camera device detects the candidate target according to the control instruction to obtain the second object feature of the candidate target.

[0019] Further, controlling the camera device deployed on the vehicle to detect the candidate target and obtaining the second object feature of the candidate target includes:

[0020] If the target warning area is the third warning area, a control instruction is sent to the camera device deployed on the vehicle, so that the camera device detects the candidate target according to the control instruction to obtain the second object feature of the candidate target.

[0021] Further, analyzing the candidate target based on the first object feature and the second object feature to obtain an analysis result includes:

[0022] Performing time synchronization calibration on the first object feature and the second object feature to obtain the calibrated first object feature and the calibrated second object feature;

[0023] Adding the calibrated first object feature and the calibrated second object feature to the vehicle coordinate system of the vehicle, and in the vehicle coordinate system, obtaining the candidate target features that match between the calibrated first object feature and the calibrated second object feature, and the similarity corresponding to the candidate target features;

[0024] Respectively calling the radar device and the camera device to track the candidate target to obtain the feature change situation of the candidate target;

[0025] Evaluating the stability of the candidate target features based on the feature change situation to obtain the tight coupling score between the radar device and the camera device;

[0026] Determining the corresponding analysis result based on the similarity and the tight coupling score.

[0027] Further, determining the corresponding analysis result based on the similarity and the tight coupling score includes:

[0028] If the similarity is greater than the preset similarity and the tight coupling score is greater than the preset score, determining that the analysis result is whether the candidate target is an abnormal target;

[0029] If the similarity is less than or equal to the preset similarity and the tight coupling score is less than or equal to the preset score, determining that the analysis result is that the candidate target is not an abnormal target.

[0030] Further, the method further includes:

[0031] Detecting the environmental features of the environment where the vehicle is currently located;

[0032] Obtaining the target information of the abnormal target, and determining a target joint defense strategy based on the target information and the environmental features;

[0033] According to the target joint defense strategy, a joint defense vehicle is determined from the environment where the vehicle is currently located, and target information of the abnormal target is sent to the joint defense vehicle to wake up the joint defense vehicle and enter the sentinel mode.

[0034] In a second aspect, an embodiment of the present invention provides a vehicle-based environment perception device, the device comprising:

[0035] A detection module, configured to, when the vehicle enters the sentinel mode, call a radar device deployed on the vehicle to detect at least one candidate target and a first object feature of the candidate target in an environment where the vehicle is located;

[0036] An acquisition module, used to acquire a target warning area where the candidate target is currently located, and determine whether a detection condition is met at a current moment based on the target warning area;

[0037] a control module, configured to control a camera device deployed on the vehicle to detect the candidate target and obtain a second object feature of the candidate target if the detection condition is met at the current moment;

[0038] An analysis module is used to analyze the candidate target based on the first object feature and the second object feature to obtain an analysis result, and perform a corresponding safety warning operation based on the analysis result, wherein the analysis result is used to characterize whether the candidate target is an abnormal target.

[0039] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0041] This application enables the collaborative work of radar devices and camera devices to achieve multi-dimensional feature acquisition of candidate targets, improve the accuracy of target analysis, and effectively reduce false alarms and false records. Secondly, the radar device is used to detect candidate targets and the first object features, and the camera device obtains the second object features. Without complex algorithms and only adjusting the UWB radar, the detection functions of the moving distance, speed, and category of objects can be achieved. Then, through the combined action of the radar device and the camera device, the defect of insufficient detection distance of the UWB radar is made up, the detection range of the surrounding environment is expanded, and the risk of intrusion by abnormal targets can be effectively alerted. In addition, the feature acquisition of the front and rear ends of the target by the radar and camera devices realizes the whole-process tightly coupled identification and monitoring of suspicious objects from the far end to the near end. Brief Description of the Drawings

[0042] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1 is a flowchart of a vehicle-based environment perception method according to some embodiments of the present invention;

[0044] Figure 2 is a working schematic diagram of the sentry mode according to some embodiments of the present invention;

[0045] Figure 3 is a flowchart of the vehicle detection range and the warning area according to some embodiments of the present invention;

[0046] Figure 4 is a flowchart of a vehicle-based environment perception method according to some embodiments of the present invention;

[0047] Figure 5 is a structural block diagram of a vehicle-based environment perception device according to an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Description of the Embodiments

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] According to an embodiment of the present invention, there is provided an environment perception method, apparatus, device, and medium based on a vehicle. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0051] In this embodiment, an environment perception method based on a vehicle is provided. Figure 1 It is a flowchart of an environment perception method based on a vehicle according to an embodiment of the present invention, as Figure 1 shown, and the process includes the following steps:

[0052] Step S101, when the vehicle enters the sentry mode, call the radar device deployed on the vehicle to detect at least one candidate target in the environment where the vehicle is located and the first object feature of the candidate target.

[0053] In the embodiment of the present application, the user operates on the mobile phone side or the vehicle-mounted device according to the requirement to select to turn on the camp sentry mode. At this time, the system receives the opening instruction and starts to prepare for the relevant operations to enter the sentry mode.

[0054] Automatically detect the various states of the vehicle to determine whether the preconditions for turning on the camp sentry are met. These preconditions include, but are not limited to, the vehicle state (such as whether it is in a parked state, etc.), the gear state (whether it is in the P gear, etc.), the battery power (the power needs to meet a certain threshold to ensure the normal operation of the sentry mode), etc. When all the preconditions are met, the subsequent operations will continue. If any one of the conditions is not met, the sentry mode will not be turned on, and the unmet condition information may be fed back to the user through the vehicle-mounted device or the mobile phone.

[0055] When it is determined that the preconditions are met, the radar device, as part of the vehicle perception module, starts to perform detection. The radar device emits electromagnetic wave signals, and the signals are reflected back after encountering the objects in the environment where the vehicle is located and are received by the radar device. By analyzing the received reflected signals, at least one candidate target in the environment where the vehicle is located is identified.

[0056] During the recognition process, relevant information of candidate targets, i.e., the first object features, is determined based on the characteristics of the signals (such as reflection intensity, time delay, etc.). Specifically, these first object features include, but are not limited to, the type of the target (such as a person, an animal, other vehicles, etc., which can be determined by matching the reflection pattern and characteristics of the signal with a preset target type model), the position distance of the target (calculated based on the time difference between signal transmission and reception and the propagation speed of electromagnetic waves), the speed of the target (calculated by analyzing the position changes of the target at multiple consecutive moments), the azimuth angle of the target (determined based on the angle information of the reflected signal received by the radar device), and other key information. The radar device outputs the above-mentioned information about the candidate target (including the first object features such as the ID, type, position distance, speed, azimuth angle of the target) detected in real time to the control module.

[0057] It should be noted that, as Figure 2 shown, the user can operate to turn on or off the sentry system through the control buttons on the mobile phone or in-vehicle device. The control module plays a core role in this. It can control the sensing module, such as millimeter-wave radar, surround-view camera, etc., to realize the opening and closing of the sensing system; at the same time, it can also control the execution module, including warning lights, warning speakers, display screens, etc., to perform functions such as light warning, sound warning, screen display, data upload, and notifying the user.

[0058] Step S102, obtain the target warning area where the candidate target is currently located, and determine whether the current moment meets the detection conditions based on the target warning area.

[0059] In the embodiment of the present application, determining whether the current moment meets the detection conditions based on the target warning area includes: detecting whether the target warning area is the first warning area; if the target warning area is the first warning area, it is determined that the current moment does not meet the detection conditions; if the target warning area is the second warning area, it is determined that the current moment meets the detection conditions; or, if the target warning area is the third warning area, it is determined that the current moment meets the detection conditions.

[0060] Specifically, the radar device transmits the monitored object information (including the position information of the object) to the control module in real time. The control module determines whether the target warning area where the object is located is the first warning area (vigilance area), the second warning area (warning area), or the third warning area (alarm area) according to the preset ranges of each warning area (parameters such as the boundary distance and angle of different areas have been preset in advance. For example, with the vehicle as the center, the area farther from the vehicle is set as the vigilance area, the area slightly closer is the warning area, and the area closer to the vehicle is the alarm area).

[0061] If the target warning area is the first warning area (vigilance area): The control module determines that the detection condition is not met at the current moment. At this time, only the relevant information of the object (such as the approximate position and moving direction of the object) is recorded, and some low-intensity operations are performed according to the preset vigilance strategy, such as making a simple record or prompt in the vehicle internal system, but the camera device is not activated for further detection.

[0062] If the target warning area is the second warning area (alert area): The control module determines that the detection condition is met at the current moment. Entering this area indicates that the object is relatively close to the vehicle, and there may be certain risks.

[0063] If the target warning area is the third warning area (alarm area): The control module determines that the detection condition is met at the current moment. This area is very close to the vehicle, and the presence of the object may pose a greater threat to the vehicle safety.

[0064] When the control module determines that the detection condition is met at the current moment (i.e., the target warning area is the alert area or the alarm area), the control module sends a start command to the camera device (such as a surround view camera, etc.) deployed on the vehicle. After receiving the command, the camera device immediately starts working and detects the candidate targets (i.e., the objects within the alert area or the alarm area).

[0065] The camera device obtains the appearance features (such as shape, color, size, etc.), position information (position coordinates relative to the vehicle, etc.) and motion state (moving speed, direction, etc.) of the candidate targets by collecting image information. These image information and relevant feature data are transmitted back to the control module in real time for subsequent analysis and processing, such as fusing with the information obtained by the radar device for judgment, further confirming the nature and threat level of the target, and taking corresponding measures according to the preset strategy (such as issuing an alarm, recording a video, etc.).

[0066] It should be noted that the first warning area, the second warning area and the third warning area are warning areas constructed with the vehicle as the center and different distances as the radius; the first warning area contains the second warning area, and the second warning area contains the third warning area. For example Figure 3As shown in the figure, four surround-view cameras are respectively located in the front, rear, left, and right directions of the vehicle body, specifically in the middle of the front grille, the middle of the rear bumper, and the left and right outer rearview mirror bases. Their sensing areas are represented in light green; five millimeter-wave radars are distributed in the middle of the front end of the vehicle and the four corners of the front and rear bumpers. The forward millimeter-wave radar is behind the middle trim cover of the front grille, two front corner radars are at the left and right side corners of the front bumper, and two rear corner radars are at the left and right side corners of the rear bumper. Their sensing areas are presented in light blue. At the same time, three warning areas are marked in the figure. From the outside to the inside, they are the first warning area (vigilance area), the second warning area (alert area), and the third warning area (alarm area), all centered on the vehicle and constructed with different distances as radii, and showing an inclusion relationship.

[0067] Step S103, if the current moment meets the detection condition, then control the camera device deployed on the vehicle to detect the candidate target, and obtain the second object feature of the candidate target.

[0068] In the embodiment of the present application, controlling the camera device deployed on the vehicle to detect the candidate target and obtain the second object feature of the candidate target includes the following steps A1 - A3:

[0069] Step A1, if the target warning area is the second warning area, then obtain the first object identifier when the candidate target enters the first warning area, and the second object identifier when the candidate target is in the second warning area.

[0070] Specifically, when the radar device detects that an object enters the second warning area (alert area), the sensing module (such as millimeter-wave radar) transmits the information related to the detected object to the control module in real time. The control module first queries the record to obtain the first object identifier recorded when the candidate target enters the first warning area (vigilance area), and at the same time obtains the second object identifier when the candidate target is in the second warning area. These identifiers are unique identification marks assigned by the system according to the characteristic information of the object when it is detected by the radar.

[0071] Step A2, compare the first object identifier and the second object identifier.

[0072] Specifically, after the control module obtains the first object identifier and the second object identifier, it starts the comparison program. Through a preset comparison algorithm, it makes a detailed comparison of these two identifiers to determine whether they are consistent in terms of characters, coding rules, etc. This comparison process is realized based on the mechanism of storing and processing data inside the system to accurately determine the matching situation of the two identifiers.

[0073] Step A3, if the first object identifier is consistent with the second object identifier, then send a control instruction to the camera device deployed on the vehicle, so that the camera device detects the candidate target according to the control instruction, and obtains the second object feature of the candidate target.

[0074] Specifically, by comparing and determining that the first object identification is consistent with the second object identification, it indicates that the object has continuously moved from the first warning area to the second warning area, and there is a potential risk. At this time, the control module sends a control instruction to the camera equipment deployed on the vehicle (such as a surround view camera installed in the front, rear, left and right directions of the outer boundary of the vehicle body). After receiving the instruction, the camera equipment starts working, detects the candidate target according to the instruction requirements, obtains the second object features such as the appearance, shape, and size of the candidate target by collecting images, and feeds these feature information back to the control module.

[0075] Step S104, analyzing the candidate target based on the first object feature and the second object feature to obtain an analysis result, and performing a corresponding safety warning operation based on the analysis result, wherein the analysis result is used to characterize whether the candidate target is an abnormal target.

[0076] In an embodiment of the present application, after the camera device acquires the second object feature of the candidate target, it transmits it to the control module. The control module simultaneously calls the first object feature detected by the radar device before, and uses a preset data analysis algorithm to perform a comprehensive comparison and analysis on the two sets of features. For example, the type, position distance, speed, appearance characteristics and other information of the target are compared to judge the consistency and change trend of each feature, so as to obtain the analysis result and determine whether the candidate target is an abnormal target. If the analysis result shows that the candidate target is an abnormal target, the control module sends instructions to the execution module (such as warning lights, warning speakers, etc.) according to the pre-set safety warning strategy, and executes the corresponding safety warning operation, such as flashing lights, sound alarms, etc.; if the analysis result shows that the candidate target is not an abnormal target, no warning operation is executed or a lower level warning operation is executed.

[0077] As an example, assuming that the vehicle is in sentry mode, the radar device first detects that an object has entered the alert area and obtains the first object feature of the object, such as the target type is initially judged to be an animal, the location is 10 meters away from the vehicle, and the speed is slow. When the object enters the alert area, the camera device starts to detect it and obtains the second object feature, such as the appearance of a large wild boar. The control module analyzes the first object feature and the second object feature and finds that the target type and other features can confirm each other and meet the abnormal target features that may pose a threat to the vehicle. The control module then sends a command to the warning speaker to sound an alarm to alert people around and send a notification message to the owner's mobile phone.

[0078] In this application, the radar device and the camera device work together to collect multi-dimensional features of candidate targets, improve the accuracy of target analysis, and effectively reduce false alarms and false records. Secondly, the radar device is used to detect candidate targets and the first object features, and the camera device obtains the second object features. Without complex algorithms and only adjusting for the UWB radar, the detection functions of the object's moving distance, speed, and category can be realized. Then, through the combined action of the radar device and the camera device, the defect of insufficient detection distance of the UWB radar is made up, the detection range of the surrounding environment is expanded, and the risk of abnormal target intrusion can be effectively alerted. In addition, the radar and camera devices collect the features of different stages of the front and rear ends of the target, realizing the whole-process tightly coupled identification and monitoring of suspicious objects from the far end to the near end.

[0079] Figure 4 is a flowchart of an environment perception method based on a vehicle according to an embodiment of the present invention, as Figure 4 shown, and the process includes the following steps:

[0080] Step S201, when the vehicle enters the sentry mode, call the radar device deployed on the vehicle to detect at least one candidate target in the environment where the vehicle is located and the first object features of the candidate target.

[0081] In the embodiment of the present application, when the user turns on the camp sentry mode through the mobile phone or the in-vehicle device, the control module of the vehicle issues an instruction to start the radar device deployed on the vehicle, such as a millimeter-wave radar. The radar device starts to work, emits electromagnetic wave signals, and these signals are reflected back after encountering objects in the surrounding environment of the vehicle. The radar device receives the reflected signals and analyzes the characteristics of the signals through the built-in signal processing algorithm. Based on the analysis results, at least one candidate target is identified, and at the same time, the first object features of the candidate target are determined, including information such as the ID, type, position distance, speed, azimuth angle, etc. of the target, and these information are transmitted to the control module in real time.

[0082] Step S202, obtain the target warning area where the candidate target is currently located, and determine whether the current moment meets the detection conditions based on the target warning area.

[0083] In the embodiment of the present application, after the control module receives the candidate target information transmitted by the radar device, it judges the target warning area where the candidate target is currently located according to the preset ranges of each warning area (such as dividing the vigilant area, the warning area, and the alarm area with the vehicle as the center). If the candidate target is in the first warning area (vigilant area), it is determined that the current moment does not meet the detection conditions; if the candidate target is in the second warning area (warning area) or the third warning area (alarm area), it is determined that the current moment meets the detection conditions.

[0084] Step S203, if the current moment meets the detection condition, then control the imaging device deployed on the vehicle to detect the candidate target, and obtain the second object feature of the candidate target.

[0085] In the embodiment of the present application, controlling the imaging device deployed on the vehicle to detect the candidate target and obtain the second object feature of the candidate target includes: if the target warning area is the third warning area, then send a control instruction to the imaging device deployed on the vehicle, so that the imaging device detects the candidate target according to the control instruction, and obtains the second object feature of the candidate target.

[0086] Specifically, if it is detected that a moving object enters the pre-set third warning area (i.e., the alarm area), the radar system will immediately transmit this information and the relevant data of the moving object (such as the ID, position distance, speed, azimuth angle, etc. of the target) to the control module. After receiving this information, the control module confirms that the target warning area is the third warning area, and then sends a control instruction to the imaging device deployed on the vehicle (such as the surround view camera installed on the outer boundary of the vehicle body, etc.). After receiving the control instruction, the imaging device detects the candidate target in the alarm area in all directions according to the instruction requirements. By collecting images, analyzing images and other operations, obtain the appearance features (such as shape, color, size, etc.), position information (accurate position coordinates relative to the vehicle, etc.) and motion state (moving speed, direction, etc.) of the candidate target, so as to obtain the second object feature of the candidate target, and feedback these feature information to the control module.

[0087] Step S204, analyze the candidate target based on the first object feature and the second object feature to obtain an analysis result, and perform corresponding safety warning operations based on the analysis result, where the analysis result is used to characterize whether the candidate target is an abnormal target.

[0088] In the embodiment of the present application, analyzing the candidate target based on the first object feature and the second object feature to obtain an analysis result includes the following steps B1 - B5:

[0089] Step B1, perform time synchronization calibration on the first object feature and the second object feature to obtain the calibrated first object feature and the calibrated second object feature.

[0090] Specifically, since there may be a difference in the time when the radar device detects the first object feature and the imaging device obtains the second object feature, time synchronization calibration is required. The control module first obtains the acquisition timestamps of the two feature data, and then according to the internal clock reference and time calibration algorithm of the system, uniformly adjusts the two timestamps, and calibrates the time information of the first object feature and the second object feature to the same time scale, so as to obtain the calibrated first object feature and the calibrated second object feature.

[0091] Step B2: Add the calibrated first object feature and the calibrated second object feature to the vehicle coordinate system of the vehicle, and in the vehicle coordinate system, obtain the candidate target features that match between the calibrated first object feature and the calibrated second object feature, as well as the similarity corresponding to the candidate target features.

[0092] Specifically, the control module adds the calibrated first object feature and the calibrated second object feature to the vehicle coordinate system according to the coordinate system parameters of the vehicle itself (including information such as the position and orientation of the vehicle). In the vehicle coordinate system, through a preset feature matching algorithm, these two sets of features are compared one by one to find the candidate target features that match, such as the position and speed of the target. At the same time, using a similarity calculation model, calculate the similarity corresponding to these matching candidate target features to measure the similarity degree of the two sets of features.

[0093] As an example, assume that the vehicle is parked in a parking lot and the radar device detects a pedestrian approaching and obtains the first object feature of the pedestrian, including information such as the position 5 meters in front of the vehicle and the speed of moving towards the vehicle at 1 meter per second; at the same time, the camera device of the vehicle also captures this pedestrian and obtains the second object feature, such as the appearance and limb movements of the pedestrian. The control module adds these two sets of features to the vehicle coordinate system according to the coordinate system parameters of the vehicle itself, such as the vehicle's current front facing east and the coordinate origin set at the center of the rear axle of the vehicle. In this coordinate system, the control module uses a preset feature matching algorithm to carefully compare the two sets of features and finds that the position of the pedestrian detected by the radar corresponds to the position of the pedestrian in the camera image, and the speed change trend is also the same. At the same time, using the similarity calculation model, substitute the matching position, speed and other features into the operation to obtain a similarity of 80%, which indicates that the similarity degree of the features of this pedestrian obtained by the radar and the camera is relatively high, further confirming that the two detect the same candidate target.

[0094] Step B3: Call the radar device and the camera device respectively to track the candidate target and obtain the change situation of the features of the candidate target.

[0095] Specifically, when it is necessary to track the candidate target, the control module sends tracking instructions to the radar device and the camera device respectively. The radar device continuously emits and receives electromagnetic wave signals, and by analyzing the changes in the reflected signals, it monitors the changes in the features such as the position, speed, and azimuth angle of the candidate target in real time. The camera device records the changes in the features such as the appearance, shape, and movement trajectory of the candidate target by continuously taking pictures, and both feedback the change situation of the candidate target features tracked by themselves to the control module.

[0096] Step B4: Evaluate the stability of candidate target features based on the feature change situation to obtain the tight coupling score between the radar device and the camera device.

[0097] Specifically, after the control module receives the change situations of the candidate target features fed back by the radar device and the camera device, it analyzes the change situations of the candidate target features under the tracking of different devices according to a pre-set stability evaluation algorithm. If the change trends of the candidate target features are relatively consistent during the tracking processes of the radar device and the camera device, it indicates high feature stability and a high tight coupling degree between the two; otherwise, it is low. Through such analysis and calculation, the tight coupling score between the radar device and the camera device is obtained.

[0098] It should be noted that first, the control module receives the data on the change situations of the candidate target features fed back by the radar device and the camera device, and these features include but are not limited to the position, speed, appearance, etc. of the candidate target. Then, according to the pre-set stability evaluation algorithm, it analyzes the change situations of the candidate target features under the tracking of different devices (i.e., the radar device and the camera device). During the analysis process, it compares the change trends of each feature of the candidate target recorded by the radar device and the camera device over time. If the change trends of the features of the candidate target are highly consistent during the tracking processes of the radar device and the camera device, such as similar change amplitudes of the position, the same speed change trend, no obvious conflicts in appearance features, etc., this indicates high stability of the candidate target features, and further indicates a high tight coupling degree between the radar device and the camera device for tracking the candidate target; otherwise, if the difference in the recorded feature change trends between the two is large, it means low feature stability and low tight coupling degree. Finally, through quantitative calculation of the analysis results of the consistency of the feature change trends, a specific value is obtained, and this value is the tight coupling score between the radar device and the camera device. Among them, the process of quantitative calculation can be to assign corresponding weights to each feature dimension according to the importance of different feature dimensions for judging the stability of the candidate target features, and calculate the tight coupling score based on the weights and the scores corresponding to the feature dimensions.

[0099] As an example, classify the received feature data according to dimensions such as position, speed, appearance, etc., and extract the key feature information from each group of data. For example, for the position information, extract the horizontal and vertical coordinates of the target in the coordinate system; for the speed information, extract the magnitude and direction of the speed; for the appearance information, extract the feature parameters such as the shape contour and color of the target.

[0100] For each feature dimension, calculate the change trends of the data fed back by the radar device and the camera device respectively. Taking the position dimension as an example, by calculating the coordinate differences of the target positions at different time points, the change amount of the position is obtained; then, according to the time interval, the change rate of the position is calculated to describe the change trend of the position. Similarly, for the speed dimension, calculate the change amounts and change rates of the speed magnitude and direction; for the appearance dimension, evaluate the degree and direction of appearance change by comparing the appearance feature parameters at different time points.

[0101] Compare the change trends of the radar device and the camera device in each feature dimension to evaluate the degree of consistency between them. A similarity calculation algorithm, such as the cosine similarity algorithm, can be used to calculate the similarity score of the change trends of the two devices in the same feature dimension. The score range is usually set between 0 and 1, and the higher the score, the more consistent the change trends. For example, if the similarity score of the change trends of the radar device and the camera device in the position dimension is 0.8, it indicates a high degree of consistency in their position changes.

[0102] According to the importance of different feature dimensions for judging the stability of the candidate target features, assign corresponding weights to each feature dimension. For example, the weight of the position dimension is set to 0.4, the weight of the speed dimension is set to 0.3, and the weight of the appearance dimension is set to 0.3. The weight assignment can be adjusted according to the actual application scenario and requirements.

[0103] Multiply the similarity score of each feature dimension by the corresponding weight, and then sum up the weighted scores of each dimension to obtain a comprehensive score. For example, the similarity score of the position dimension is 0.8 and the weight is 0.4; the similarity score of the speed dimension is 0.7 and the weight is 0.3; the similarity score of the appearance dimension is 0.6 and the weight is 0.3. Then the comprehensive score = 0.8×0.4 + 0.7×0.3 + 0.6×0.3 = 0.32 + 0.21 + 0.18 = 0.71.

[0104] Step B5, determine the corresponding analysis result based on the similarity and the tight coupling score.

[0105] Specifically, determine the corresponding analysis result based on the similarity and the tight coupling score, including: if the similarity is greater than the preset similarity and the tight coupling score is greater than the preset score, then determine whether the candidate target is an abnormal target as the analysis result; if the similarity is less than or equal to the preset similarity and the tight coupling score is less than or equal to the preset score, then determine that the candidate target is not an abnormal target as the analysis result.

[0106] It should be noted that if the similarity is greater than the preset similarity, it indicates that the feature consistency of the candidate target under different device detections is relatively high, and the tight coupling score is greater than the preset score, which means that the collaborative tightness between the radar device and the camera device in tracking the candidate target is good. When both of these conditions are met, the control module will determine that the analysis result is that the candidate target is an abnormal target. On the contrary, if the similarity is less than or equal to the preset similarity, it means that the feature consistency of the candidate target under different device detections is poor, and the tight coupling score is less than or equal to the preset score, that is, the collaborative tightness between the radar device and the camera device in tracking is not good. When both of these conditions are simultaneously established, the control module will determine that the analysis result is that the candidate target is not an abnormal target.

[0107] In the embodiment of the present application, the method further includes the following steps C1 - C3:

[0108] Step C1, detecting the environmental characteristics of the current environment where the vehicle is located.

[0109] In the embodiment of the present application, various sensors equipped on the vehicle start to work. For example, the camera can identify information such as the type of objects in the environment (such as buildings, trees, other vehicles, etc.), road conditions (such as whether the road surface is flat and whether there are obstacles), weather conditions (such as sunny, rainy, snowy, etc.) by taking pictures of the surrounding scenes; millimeter-wave radars and ultrasonic radars can detect data such as the distance, position, speed, and direction of surrounding objects to perceive the spatial layout and object distribution around the vehicle; in addition, the vehicle can also obtain its own geographical location information through the Global Positioning System (GPS), and combine with map data to understand environmental characteristics such as the topography and surrounding facilities of the area where it is located. The data collected by different sensors will be transmitted to the vehicle's control module in real time, and the control module integrates and analyzes the data to comprehensively detect and determine the environmental characteristics of the current environment where the vehicle is located.

[0110] Step C2, obtaining the target information of the abnormal target, and determining the target joint defense strategy based on the target information and the environmental characteristics.

[0111] In the embodiment of the present application, when the vehicle's perception system (such as cameras, radars, etc.) detects an abnormal target (such as a suspicious person approaching, an unknown object approaching quickly, etc.), it will immediately track and analyze the target to obtain the target information of the abnormal target, including but not limited to the type of the target (whether it is a person, an animal, or an object), the speed of the target, the movement trajectory of the target, the appearance characteristics of the target, etc. At the same time, the control module will call the environmental characteristic data of the current environment where the vehicle is located that has been detected.

[0112] Then, the control module will comprehensively analyze the target information and environmental characteristics according to preset algorithms and rules. For example, if the abnormal target is a fast-moving object and the surrounding environment is relatively open, a joint defense strategy of rapid warning and expanded monitoring range will be determined; if the abnormal target is a suspicious person and there are other vehicles parked around the vehicle, a joint defense strategy of jointly monitoring and alarming with the surrounding vehicles may be determined. Through such comprehensive analysis and judgment, a suitable target joint defense strategy is finally determined.

[0113] Step C3: Determine joint defense vehicles from the environment where the vehicle is currently located according to the target joint defense strategy, and send the target information of the abnormal target to the joint defense vehicles to wake up the joint defense vehicles to enter the sentry mode.

[0114] In the embodiment of the present application, information such as whether the surrounding vehicles support the joint defense function, the distance and position relationship with the vehicle itself is detected. According to this information, joint defense vehicles that meet the requirements of the joint defense strategy are determined from the surrounding vehicles. Then, the control module sends the target information of the abnormal target to the determined joint defense vehicles through wireless communication, including detailed data such as the type, speed, and movement trajectory of the target. After receiving this information, the vehicle control system of the joint defense vehicle will automatically identify and wake up the vehicle to enter the sentry mode. In the sentry mode, the perception systems (such as cameras, radars, etc.) of the joint defense vehicles will be activated and start real-time monitoring of the surrounding environment, work together with the vehicle itself to jointly deal with the abnormal target, and achieve the purpose of joint defense.

[0115] It should be noted that taking a parking lot scenario as an example, when the vehicle detects an abnormal target (such as a suspicious person wandering), the target joint defense strategy is first determined. Common joint defense strategies include the area warning strategy, that is, strengthening the monitoring within a certain range around the area where the abnormal target appears; the relay tracking strategy, if the abnormal target moves, each vehicle relays to track and monitor it; the collaborative alarm strategy, once an abnormality is confirmed, multiple vehicles sound the alarm simultaneously to deter. After determining the joint defense strategy, the joint defense vehicles are determined. The vehicle first obtains the status information of the surrounding vehicles, including whether they are in a state of being able to participate in joint defense, vehicle battery power, signal strength, etc. If it is the area warning strategy, vehicles within a certain distance (such as within 50 meters) around the area where the abnormal target appears and in a state of being able to participate in joint defense are preferentially selected; if it is the relay tracking strategy, vehicles within a certain range (such as within 100 meters ahead) in the moving direction of the abnormal target and with good tracking capabilities (such as equipped with high-precision radars and cameras) are selected; for the collaborative alarm strategy, all vehicles that can participate in joint defense within a certain range (such as within 30 meters) around are selected. After determining the joint defense vehicles, the target information of the abnormal target, such as the position, appearance characteristics, and action trajectory of the target, is sent to these vehicles through wireless communication technology, thereby waking up the joint defense vehicles to enter the sentry mode.

[0116] The method provided in the embodiment of the present application can realize the monitoring and prevention of abnormal targets by multiple vehicles in coordination by determining the joint defense vehicle and waking it up to enter the sentry mode, thereby expanding the monitoring range, improving the ability to perceive potential threats, and being able to promptly detect and respond to abnormal situations such as theft and destruction, thereby enhancing the safety of vehicles when parked. From the perspective of resource utilization, with the help of vehicle networking technology, joint defense vehicles can be selected on demand in the environment where the vehicle is located, avoiding all vehicles being in a state of alert for a long time, effectively saving energy and equipment loss, and improving resource utilization efficiency. From the perspective of user experience, it provides car owners with more reliable vehicle protection guarantees and reduces concerns about risks encountered by vehicles during parking.

[0117] In this embodiment, a vehicle-based environment perception device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0118] This embodiment provides a vehicle-based environment perception device, such as Figure 5 As shown, including:

[0119] A detection module 501 is used to call a radar device deployed in the vehicle to detect at least one candidate target and a first object feature of the candidate target in the environment where the vehicle is located when the vehicle enters the sentry mode;

[0120] An acquisition module 502 is used to acquire a target warning area where the candidate target is currently located, and determine whether a detection condition is met at the current moment based on the target warning area;

[0121] The control module 503 is used to control the camera device deployed on the vehicle to detect the candidate target and obtain the second object feature of the candidate target if the detection condition is met at the current moment;

[0122] The analysis module 504 is used to analyze the candidate target based on the first object feature and the second object feature, obtain an analysis result, and perform a corresponding safety warning operation based on the analysis result, wherein the analysis result is used to characterize whether the candidate target is an abnormal target.

[0123] In an embodiment of the present application, the acquisition module 502 is specifically configured to detect whether the target warning area is a first warning area; if the target warning area is the first warning area, it is determined that the current time does not meet the detection condition; if the target warning area is the second warning area, it is determined that the current time meets the detection condition; or, if the target warning area is the third warning area, it is determined that the current time meets the detection condition; wherein, the first warning area, the second warning area, and the third warning area are warning areas constructed with the vehicle as the center and different distances as the radii; the first warning area includes the second warning area, and the second warning area includes the third warning area.

[0124] In an embodiment of the present application, the control module 503 is specifically configured to, if the target warning area is the second warning area, obtain the first object identifier when the candidate target enters the first warning area and the second object identifier when the candidate target is in the second warning area; compare the first object identifier and the second object identifier; if the first object identifier is consistent with the second object identifier, send a control instruction to the camera device deployed on the vehicle, so that the camera device detects the candidate target according to the control instruction to obtain the second object feature of the candidate target.

[0125] In an embodiment of the present application, the control module 503 is specifically configured to, if the target warning area is the third warning area, send a control instruction to the camera device deployed on the vehicle, so that the camera device detects the candidate target according to the control instruction to obtain the second object feature of the candidate target.

[0126] In an embodiment of the present application, the analysis module 504 is specifically configured to perform time synchronization calibration on the first object feature and the second object feature to obtain the calibrated first object feature and the calibrated second object feature; add the calibrated first object feature and the calibrated second object feature to the vehicle coordinate system of the vehicle, and in the vehicle coordinate system, obtain the candidate target features that match between the calibrated first object feature and the calibrated second object feature, and the similarity corresponding to the candidate target features; respectively call the radar device and the camera device to track the candidate target to obtain the feature change situation of the candidate target; evaluate the stability of the candidate target features based on the feature change situation to obtain the tight coupling score between the radar device and the camera device; determine the corresponding analysis result based on the similarity and the tight coupling score.

[0127] In an embodiment of the present application, the analysis module 504 is specifically configured to, if the similarity is greater than the preset similarity and the tight coupling score is greater than the preset score, determine that the analysis result is whether the candidate target is an abnormal target; if the similarity is less than or equal to the preset similarity and the tight coupling score is less than or equal to the preset score, determine that the analysis result is that the candidate target is not an abnormal target.

[0128] In the embodiment of the present application, the device further includes: a processing module, configured to detect environmental characteristics of the current environment where the vehicle is located; obtain target information of an abnormal target, and determine a target joint defense strategy based on the target information and the environmental characteristics; determine joint defense vehicles from the current environment where the vehicle is located according to the target joint defense strategy, and send the target information of the abnormal target to the joint defense vehicles to wake up the joint defense vehicles to enter the sentry mode.

[0129] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As Figure 6 shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common main board or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional implementation manners, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system).

[0130] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0131] Among them, the memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0132] The memory 20 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of a computer device for the display of a kind of mini-program landing page, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely provided relative to the processor 10, and these remote memories may be connected to the computer device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memory.

[0134] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0135] The embodiments of the present invention also provide a computer-readable storage medium. The methods according to the embodiments of the present invention may be implemented in hardware, firmware, or may be implemented as computer code that can be recorded on a storage medium, or may be implemented as computer code originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and to be stored in a local storage medium, so that the methods described herein may be stored in such software processes on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium may further include a combination of the above types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods shown in the above embodiments are implemented.

[0136] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A vehicle-based environmental perception method, characterized in that, The method includes: When the vehicle enters the sentinel mode, calling a radar device deployed on the vehicle to detect at least one candidate target in the environment where the vehicle is located and a first object feature of the candidate target; Obtaining a target warning area where the candidate target is currently located, and determining whether the detection condition is satisfied at the current moment based on the target warning area; If the detection condition is satisfied at the current moment, controlling a camera device deployed on the vehicle to detect the candidate target to obtain a second object feature of the candidate target; Analyzing the candidate target based on the first object feature and the second object feature to obtain an analysis result, and performing a corresponding safety warning operation based on the analysis result, where the analysis result is used to indicate whether the candidate target is an abnormal target.

2. The method according to claim 1, wherein The determining whether the detection condition is satisfied at the current moment based on the target warning area includes: Detecting whether the target warning area is a first warning area; If the target warning area is the first warning area, determining that the detection condition is not satisfied at the current moment; If the target warning area is a second warning area, determining that the detection condition is satisfied at the current moment; or, if the target warning area is a third warning area, determining that the detection condition is satisfied at the current moment; Wherein, the first warning area, the second warning area, and the third warning area are warning areas constructed with the vehicle as the center and different distances as the radii; the first warning area contains the second warning area, and the second warning area contains the third warning area.

3. The method according to claim 1, wherein The controlling a camera device deployed on the vehicle to detect the candidate target to obtain a second object feature of the candidate target includes: If the target warning area is the second warning area, obtaining a first object identifier when the candidate target enters the first warning area and a second object identifier when the candidate target is in the second warning area; Comparing the first object identifier and the second object identifier; If the first object identifier is the same as the second object identifier, sending a control instruction to the camera device deployed on the vehicle so that the camera device detects the candidate target according to the control instruction to obtain a second object feature of the candidate target.

4. The method according to claim 1, characterized in that, The controlling a camera device deployed on the vehicle to detect the candidate target to obtain a second object feature of the candidate target includes: If the target warning area is the third warning area, sending a control instruction to the camera device deployed on the vehicle so that the camera device detects the candidate target according to the control instruction to obtain a second object feature of the candidate target.

5. The method according to claim 1, wherein The analyzing the candidate target based on the first object feature and the second object feature to obtain an analysis result includes: Performing time synchronization calibration on the first object feature and the second object feature to obtain a calibrated first object feature and a calibrated second object feature; Add the calibrated first object feature and the calibrated second object feature to the vehicle coordinate system of the vehicle, and in the vehicle coordinate system, obtain the candidate target features that match between the calibrated first object feature and the calibrated second object feature, and the similarity corresponding to the candidate target features; Call the radar device and the camera device respectively to track the candidate target, and obtain the feature change situation of the candidate target; Evaluate the stability of the candidate target features based on the feature change situation, and obtain the tight coupling score between the radar device and the camera device; Determine the corresponding analysis result based on the similarity and the tight coupling score.

6. The method according to claim 5, wherein The determining the corresponding analysis result based on the similarity and the tight coupling score includes: If the similarity is greater than the preset similarity and the tight coupling score is greater than the preset score, determine whether the analysis result is that the candidate target is an abnormal target; If the similarity is less than or equal to the preset similarity and the tight coupling score is less than or equal to the preset score, determine that the analysis result is that the candidate target is not an abnormal target.

7. The method according to claim 1, characterized in that The method further includes: Detect the environmental features of the current environment where the vehicle is located; Obtain the target information of the abnormal target, and determine the target joint defense strategy based on the target information and the environmental features; Determine the joint defense vehicle from the current environment where the vehicle is located according to the target joint defense strategy, and send the target information of the abnormal target to the joint defense vehicle to wake up the joint defense vehicle to enter the sentry mode.

8. A vehicle-based environmental perception device, characterized in that, The device includes: A detection module, configured to, when the vehicle enters the sentry mode, call a radar device deployed on the vehicle to detect at least one candidate target and the first object feature of the candidate target in the environment where the vehicle is located; An acquisition module, configured to acquire the target warning area where the candidate target is currently located, and determine whether the current moment meets the detection condition based on the target warning area; A control module, configured to, if the current moment meets the detection condition, control a camera device deployed on the vehicle to detect the candidate target, and obtain the second object feature of the candidate target; An analysis module, configured to analyze the candidate target based on the first object feature and the second object feature, obtain an analysis result, and perform a corresponding security warning operation based on the analysis result, where the analysis result is used to characterize whether the candidate target is an abnormal target.

9. A computer device, characterized in that, Includes: A memory and a processor, which are communicatively connected to each other, where the memory stores computer instructions, and the processor executes the computer instructions to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Computer instructions are stored on the computer-readable storage medium, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images