Risk identification method, system, device, vehicle and electronic device

Abstract

The present application relates to a kind of risk identification method, system, device, vehicle and electronic equipment, the method comprises: by first model, at least one sensor data is based, generate first risk information;Determine processing granularity based on first risk level;Based on processing granularity, video data is encoded and processed, and code stream data is generated;Send early warning data to electronic equipment, to make electronic equipment extract the feature of early warning data, generate early warning feature, by second model, based on early warning feature, generate second risk information, send second risk information to vehicle, early warning data includes code stream data.In the present application, vehicle can trigger electronic equipment to determine the risk information of vehicle again by first risk level, and electronic equipment does not need to decode the code stream data obtained after encoding, so as to improve accuracy and real-time and reduce power consumption, can improve the risk identification effect of vehicle.

Description

Technical Field

[0001] This invention relates to the field of automotive safety technology, specifically to a risk identification method, system, device, vehicle, and electronic device. Background Technology

[0002] In related technologies, the in-vehicle sentry mode is used to monitor vehicle safety. It can continuously monitor the surrounding environment when the vehicle is parked, identify parking risks, and promptly alert when anomalies are detected, providing all-weather safety for the car owner. However, when monitoring parking risks through sentry mode, it often cannot simultaneously possess the following advantages: low power consumption, high accuracy, and high real-time performance, resulting in poor risk identification performance for vehicles in related technologies. Summary of the Invention

[0003] One objective of this invention is to provide a risk identification method to solve the problem of poor risk identification effect for vehicles in related technologies; a second objective is to provide a risk identification method; a third objective is to provide a risk identification system; a fourth objective is to provide a risk identification device; a fifth objective is to provide a risk identification device; a sixth objective is to provide a vehicle; and a seventh objective is to provide an electronic device.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect, embodiments of this application provide a risk identification method, including: When the risk identification mode is enabled, at least one sensor data is acquired; wherein, the at least one sensor data includes video data; Based on the data from the at least one sensor, a first risk information is generated using a first model; wherein the first risk information is used to indicate a first risk level. If the first risk level meets the first risk condition, the processing granularity is determined based on the first risk level. Based on the processing granularity, the video data is encoded to generate bitstream data; The warning data is sent to an electronic device, which extracts features from the warning data to generate warning features. Based on the warning features, a second model is used to generate second risk information, which is then sent to the vehicle. The warning data includes the code stream data, and the accuracy performance of the first model is lower than that of the second model.

[0005] In some embodiments, generating first risk information based on the at least one sensor data using a first model includes: For each of the sensor data, a sensor feature is generated; wherein the video data is used to indicate the target object; The features from multiple sensors are fused to generate a fused feature; Based on the fusion features, a recognition result is generated; wherein the recognition result is used to indicate the distance between the target object and the vehicle, and the distance is used to indicate the recognition accuracy; Based on the identification results, the first risk information is generated.

[0006] In other words, the accuracy of object recognition can be determined based on distance, thereby reducing the information required for recognizing some distant objects and accelerating the generation of the first risk information.

[0007] In some embodiments, the method further includes: Receive second risk information sent by the electronic device; wherein the second risk information is used to indicate a second risk level; When the second risk level is the first preset level, in response to the vehicle power-on command, a first prompt message is generated based on the second risk information; When the second risk level is the second preset level, a second prompt message is generated based on the second risk information and sent to the user device; The second preset level is greater than the first preset level.

[0008] In other words, when the risk is high, the user can be notified in a timely manner; when the risk is moderate, the user can be prompted to check the vehicle after boarding. This tiered notification strategy ensures that the user does not have to check the vehicle immediately after receiving each notification, and also prevents the user from forgetting to check the vehicle.

[0009] In some embodiments, the method further includes: If the storage conditions are not met at the first risk level, or if the storage conditions are not met at the second risk level, the storage area corresponding to the video data is determined as the first storage area. If the second risk level meets the storage conditions, determine the second storage area corresponding to the video data; Wherein, the first storage duration corresponding to the first storage area is less than the second storage duration corresponding to the second storage area.

[0010] In other words, a tiered storage strategy can be used to store high-risk video data for extended periods, preventing it from being overwritten by subsequent video data; and to allow low-risk video data to reuse the first storage area, thus preventing video data from occupying storage capacity for extended periods and reducing storage costs.

[0011] Secondly, embodiments of this application provide a risk identification method, including: The system receives warning data sent by a vehicle; wherein the warning data includes bitstream data, which includes data from at least one sensor acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; a first risk information is generated based on the at least one sensor data using a first model; if the first risk level indicated by the first risk information meets a first risk condition, a processing granularity is determined based on the first risk level; and data is generated after encoding the video data based on the processing granularity. Features are extracted from the warning data to generate warning features; wherein, the warning features include bitstream features, which are features extracted from the bitstream data; Based on the warning features, a second risk information is generated using a second model; wherein the accuracy performance of the first model is lower than that of the second model. The second risk information is sent to the vehicle.

[0012] In some embodiments, the video data includes keyframes and forward prediction frames, the bitstream data includes first coded data corresponding to the keyframes and second coded data corresponding to the forward prediction frames, the second coded data includes motion vector data and residual data, and the warning feature includes coded features; the step of extracting features from the warning data to generate warning features includes: Spatial features are extracted from the first encoded data to generate spatial features; Features are extracted from the motion vector data to generate motion features, and features are extracted from the residual data to generate residual features; The motion features and the residual features are concatenated to generate concatenated features; The spatial features and the splicing features are spliced ​​together to generate the encoded features.

[0013] In other words, based on the fact that the data corresponding to I-frames in the bitstream data represents the complete frame and the data corresponding to P-frames represents the difference between the two frames, spatial structure features are extracted from I-frames as the image reference, and motion vectors and residual data are extracted from P-frames to form motion features and detail features. Finally, the features are concatenated to obtain the encoded features that characterize the video. Thus, feature extraction is performed directly on the encoded video data without decoding, avoiding the time occupied by decoding.

[0014] In some embodiments, the warning data further includes first sensor data; the step of extracting features from the warning data to generate warning features includes: The features of the first sensor data are extracted to generate the first feature; The first feature and the encoded feature are fused to generate the warning feature.

[0015] In other words, the ability to fuse multimodal data enhances the richness of early warning features, thereby improving the accuracy of secondary risk information.

[0016] In some embodiments, the second risk information is used to indicate a second risk level; sending the second risk information to the vehicle includes: If the second risk level is less than or equal to the second preset level, the second risk information will be sent to the vehicle; or, If the second risk level is less than or equal to the first preset level, the second risk information is sent to the vehicle; if the second risk level is the second preset level, the second risk information is sent to the vehicle, and the third prompt information is sent to the user device. The second preset level is greater than the first preset level.

[0017] In other words, the electronic device is used to regenerate risk information and can send the second risk information to the vehicle under any second risk level, thereby reducing the energy consumption of the electronic device; or, the electronic device can determine whether to send a prompt to the user based on the second risk level, and can send a prompt to the user when the risk level is high, thereby prompting the user more quickly.

[0018] In some embodiments, the method further includes: Based on the idle time being greater than the duration threshold, the electronic device is determined to be in sleep mode. The idle time is used to indicate that the electronic device is idle.

[0019] In other words, if the electronic device does not receive the next warning information within a certain period of time after sending the second risk information of the previous warning to the vehicle and / or sending the third prompt information to the user device, it will enter a sleep state, thereby reducing the energy consumption of the electronic device.

[0020] Thirdly, embodiments of this application provide a risk identification system, including: A vehicle is configured to acquire at least one sensor data when a risk identification mode is activated; wherein the at least one sensor data includes video data; generate first risk information based on the at least one sensor data using a first model; wherein the first risk information indicates a first risk level; determine a processing granularity based on the first risk level when the first risk level meets a first risk condition; encode the video data based on the processing granularity to generate a bitstream data; and send warning data to an electronic device; wherein the warning data includes the bitstream data. The electronic device is used to receive warning data sent by a vehicle; extract features from the warning data to generate warning features; wherein the warning features include bitstream features, which are features extracted from the bitstream data; generate second risk information based on the warning features using a second model; wherein the accuracy performance of the first model is lower than that of the second model; and send the second risk information to the vehicle.

[0021] Fourthly, embodiments of this application provide a risk identification device, including: The acquisition module is used to acquire at least one sensor data when the risk identification mode is enabled; wherein, the at least one sensor data includes video data. A first generation module is configured to generate first risk information based on the data from the at least one sensor using a first model; wherein the first risk information is used to indicate a first risk level. The first determining module is used to determine the processing granularity based on the first risk level, provided that the first risk level meets the first risk condition. The second generation module is used to encode the video data based on the processing granularity to generate bitstream data; The first sending module is used to send the warning data to the electronic device, so that the electronic device can extract the features of the warning data, generate warning features, generate second risk information based on the warning features through the second model, and send the second risk information to the vehicle. The warning data includes the code stream data, and the accuracy performance of the first model is lower than that of the second model.

[0022] Fifthly, embodiments of this application provide a risk identification device, comprising: A receiving module is configured to receive warning data sent by a vehicle; wherein the warning data includes bitstream data, the bitstream data including at least one sensor data acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; a first risk information is generated based on the at least one sensor data using a first model; if the first risk level indicated by the first risk information meets a first risk condition, a processing granularity is determined based on the first risk level; and data is generated after encoding the video data based on the processing granularity. The third generation module is used to extract features from the warning data and generate warning features; wherein, the warning features include bitstream features, which are features extracted from the bitstream data; The fourth generation module is used to generate second risk information based on the warning features using the second model; wherein the accuracy performance of the first model is lower than that of the second model. The second sending module is used to send the second risk information to the vehicle.

[0023] In a sixth aspect, embodiments of this application provide a vehicle including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the method described in the first aspect.

[0024] In a seventh aspect, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the method of the second aspect described above.

[0025] Eighthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0026] Ninthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement some or all of the steps in the above-described method.

[0027] The beneficial effects of this invention are: The vehicle can run a first model, and the electronic equipment can run a second model. The accuracy of the first model is lower than that of the second model. As a result, the vehicle can quickly determine the risk level of its environment using the first model to obtain a first risk level, thus reducing the power consumption of the vehicle in determining the risk level. In order to improve the accuracy of risk identification, when the vehicle determines that the risk level is high using the first model, it sends the warning data to the electronic equipment. As a result, the electronic equipment can obtain a second risk information with higher accuracy using the second model with higher accuracy, thus improving the accuracy of risk identification. The vehicle can encode video data by processing granularity to obtain bitstream data; the processing granularity can change with the change of the first risk level, so that the vehicle can adjust the granularity of encoding video data according to the first risk level, so that the video data obtained by different processing granularities are different, and thus the processing accuracy of the second model can be adjusted by processing granularity. Furthermore, when electronic devices extract features from the bitstream data, they do not need to decode the bitstream data, thereby avoiding the time spent decoding the bitstream data, reducing the power consumption of the electronic device, increasing the rate of generating second risk information, and improving the real-time performance of risk identification of the vehicle's environment. Attached Figure Description

[0028] Figure 1 A flowchart illustrating a risk identification method provided in this application embodiment; Figure 2 A flowchart illustrating another risk identification method provided in this application embodiment; Figure 3 A flowchart of feature extraction provided in an embodiment of this application; Figure 4 A schematic diagram of a risk identification system provided in an embodiment of this application; Figure 5 A signaling diagram for risk identification provided in an embodiment of this application; Figure 6 A schematic diagram of a risk identification architecture provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a risk identification device provided in an embodiment of this application; Figure 8 This is a schematic diagram of another risk identification device provided in an embodiment of this application; Figure 9 A schematic diagram of the composition structure of a vehicle provided in an embodiment of this application; Figure 10 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0030] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0031] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0034] In one related technology, a sentinel system acquires perception data of the surrounding environment, identifies potential security threats based on the perception data, and performs threat behavior identification processing on the perception data based on the running model to obtain identification results; the vehicle is controlled in sentinel mode to execute alarm operations according to the alarm level corresponding to the identification results. However, if the sentinel system runs a low-power small model, it will result in a high false alarm rate; if a high-power large model is used, it will result in slow inference speed and excessive power consumption, making it unable to run for a long time.

[0035] In another related technology, the vehicle acquires perception data and uploads it to the cloud; the cloud identifies potential security threats and performs threat behavior identification processing on the perception data to obtain the identification results. However, the transmission and decoding time of the perception data is long, resulting in a long latency in cloud processing.

[0036] Analysis of the aforementioned patents and other existing technologies revealed the following main problems with vehicle-mounted sentry systems and large-scale video processing technologies: 1) Difficulty in balancing power consumption and accuracy: Traditional vehicle-mounted sentry systems mostly adopt a single model architecture. If a lightweight small model is used, although the power consumption is low, it can only perform simple target detection and classification, and cannot accurately identify complex threat behaviors such as smashing, scratching, and colliding with cars. If a large model is used for deep behavioral analysis, although the accuracy is high, it needs to run continuously, resulting in high power consumption and making it unsuitable for long-term use.

[0037] 2) The bottleneck of computing power in automotive chips limits the deployment of large-scale models: Although the main chip installed in vehicles has a certain amount of AI computing power, it is mainly used for core functions such as vehicle-machine interaction, navigation, and entertainment. It cannot reserve enough computing power to run large-scale video understanding models with large parameters for a long time. Upgrading the main chip in vehicles to deploy large-scale models would significantly increase the cost of the entire vehicle. Moreover, the chip upgrade cycle is tied to the vehicle replacement cycle, and some types of vehicles cannot quickly iterate their AI capabilities.

[0038] 3) Redundant video processing flow and serious waste of computing resources: Large models often follow the traditional process of "video decoding → frame-by-frame extraction → image preprocessing → large model inference". Among them, the video decoding step requires decoding the encoded video data into the original RGB image. This process is not only time-consuming, but also requires a lot of central processing unit (CPU) / graphics processing unit (GPU) computing resources and memory bandwidth, which significantly increases the power consumption and latency of the system.

[0039] To solve the above problems, Figure 1 A flowchart of a risk identification method provided in this application embodiment is shown below. Figure 1 As shown, the method includes steps 101 to 105, wherein: Step 101: With the risk identification mode enabled, acquire at least one sensor data; wherein, the at least one sensor data includes video data.

[0040] In this embodiment of the disclosure, the vehicle includes at least one sensor for acquiring images, vehicle vibration data, radar data, etc. For example, the sensor includes at least one of the following: a camera, millimeter-wave radar, ultrasonic radar, vibration sensor, etc. The camera can be used to acquire images inside and / or outside the vehicle; multiple frames of acquired images constitute a video. Both millimeter-wave radar and ultrasonic radar can be used to acquire radar data, which is used to characterize the distance between objects around the vehicle and the vehicle itself; the vibration sensor can be used to acquire vibration data such as the time and magnitude of vehicle vibrations.

[0041] The vehicle includes sensors for acquiring image frames. For example, the vehicle includes a camera for capturing the scene around the vehicle body and obtaining image frames. Video data is obtained from multiple panoramic image frames. The video data may include video, which consists of multiple image frames. That is, multiple sensor data include video data, as well as at least one of the following: radar data, vibration data, etc.

[0042] The risk identification mode can be referred to as sentry mode. A vehicle may have a sentry mode. When the sentry function is enabled, the vehicle executes the risk identification method; when the sentry mode is disabled, the vehicle does not execute the risk identification method. Before step 101, the vehicle may, in response to a user-input activation operation, set the state corresponding to the risk identification mode to the enabled state, or the state corresponding to the risk identification mode may be set to the enabled state by default.

[0043] Step 102: Using the first model, generate first risk information based on data from at least one sensor; wherein the first risk information is used to indicate the first risk level.

[0044] In this embodiment, the vehicle possesses lightweight computing power, capable of running a first model. This first model is a small model, requiring minimal computing and storage resources to obtain the first risk information, thereby accelerating the generation of risk information and improving real-time performance while reducing energy consumption. The vehicle may include an execution module, which can be used to execute... Figure 1 The steps of the risk identification method shown include, for example, an execution module that includes a main chip, which enables the deployment of a small model on the main chip. The small model requires less computing and storage resources, does not require hardware upgrades to the main chip, and does not affect the main chip's ability to perform the vehicle's core functions.

[0045] The first model can be a model capable of processing multimodal data, thereby enabling the synthesis of sensor data from multiple types of sensors—each collecting different types of data—to obtain first risk information, even when the vehicle has multiple types of sensors. These multiple types of sensors may include image sensors, sensors that measure distances to objects around the vehicle, and sensors that measure vehicle vibrations, etc. Alternatively, the first model can be a model capable of processing video data, thus enabling the processing of video data.

[0046] The first risk information may include at least one of the following: the first risk level, the type of the first risk scenario in which the vehicle is located, and the time when the risk occurs, wherein the type of the first risk scenario may correspond to the first risk level. For example, based on data from at least one sensor, if a vehicle determines that objects around it include other vehicles, and assuming that the distance between the other vehicles and the vehicle is greater than 3 meters, then the vehicle is considered to be in a low-risk scenario, with a low risk of damage and a low first risk level; if the distance between the other vehicles and the vehicle is less than 3 meters, then the vehicle is considered to be in a medium-risk scenario, with a medium first risk level; if the distance between the other vehicles and the vehicle is less than 3 meters, and the other vehicles are moving at a speed of 60 kilometers per hour, then the vehicle is considered to be in a high-risk scenario, with a high risk of damage and a high first risk level.

[0047] Step 103: If the first risk level meets the first risk condition, determine the processing granularity based on the first risk level.

[0048] In this embodiment of the disclosure, the first risk condition is used to indicate that when the first risk level reaches the warning level, the electronic device is triggered to generate risk information again. For example, the risk level can be divided into low, medium, and high levels, and the first risk condition is used to indicate that when the first risk level is medium or high, the electronic device is triggered to generate risk information again.

[0049] The first risk condition is used to indicate that if the first risk level is met, the risk of vehicle damage is medium or high. However, due to the low computational resources of the first model of vehicle operation, i.e. the limited reliability of the first risk information, it is necessary to send at least one sensor information to the electronic device so that the electronic device can re-determine the risk of vehicle damage.

[0050] Before transmitting at least one sensor information to an electronic device, the processing granularity for the video data can be determined. The processing granularity may include the target video coding standard and the coding requirements corresponding to the target video coding standard.

[0051] For example, video coding standards may include H.264, H.265, H.266, etc., and vehicles can determine the target video coding standard from multiple video coding standards based on the first risk level.

[0052] Encoding requirements may include the number of keyframes (I-frames), the number of forward prediction frames (P-frames), and the size of macroblocks in P-frames. The bitstream data includes I-frame encoded data and P-frame encoded data; I-frame encoded data refers to the data obtained after encoding I-frames; P-frame encoded data refers to the data obtained after encoding P-frames.

[0053] Video data can include multiple image frames, each of which can be an I-frame or a P-frame. An I-frame is a frame that can be encoded independently without referencing other frames, while a P-frame requires referencing the previous frame and is encoded through motion estimation and compensation. The previous frame of a P-frame can be either an I-frame or a P-frame. For example, in a video captured by a camera showing a pedestrian walking, the pedestrian is constantly moving while the background behind them remains stationary. The first image frame in the video can be selected as the I-frame, and the next 10 frames as P-frames. The encoded I-frame is used to fully describe the first image frame, the first encoded P-frame is used to indicate the difference between the second and first image frames, and the second P-frame is used to indicate the difference between the third and second image frames. This allows for encoding the video data at a processing granularity, reducing the size of the encoded video data. The size of the bitstream data can be smaller than the size of the video data.

[0054] Before encoding the video data, the video data can be updated. For example, the video data includes first video data. The duration of the first video data input to the first model is 10 seconds. When it is determined that the first risk level is high, historical video data can be obtained. Based on the historical video data and the first video data, second video data is obtained. The duration of the second video data is 15 seconds. Thus, the second video data is encoded during the encoding process.

[0055] The processing granularity is related to the first risk level; the granularity changes with the first risk level, with finer granularity required for a higher risk level. A mapping relationship can exist between processing granularity and the first risk level, allowing the processing granularity to be determined based on the first risk level. For example, the processing requirements for H.265 might include 5 I-frames per second or 10 I-frames per second. If the first risk level indicates high risk, the processing granularity could include H.265 as the video coding standard and 10 I-frames per second; if the first risk level indicates medium risk, the processing granularity could include H.265 as the video coding standard and 5 I-frames per second.

[0056] The vehicle can determine the processing granularity of video data based on a first risk level. This allows for increased processing granularity at higher risk levels and decreased granularity at lower risk levels. Higher granularity yields more detailed information in the resulting bitstream data, leading to higher processing accuracy and more complete feature extraction. Conversely, lower granularity results in coarser information and reduced processing accuracy. Furthermore, the determined processing granularity must meet the minimum recognition accuracy of the second model, thus adaptively adjusting the recognition accuracy of the second model according to the first risk level.

[0057] Step 104: Based on the processing granularity, encode the video data to generate bitstream data.

[0058] In this embodiment of the disclosure, the bitstream data may include I-frame encoded data and P-frame encoded data arranged in time sequence, wherein the I-frame encoded data is obtained by encoding I-frames, and the P-frame encoded data is obtained by encoding P-frames; the time sequence of the I-frame encoded data in the bitstream information is determined by the time sequence of the I-frames in the video data; the time sequence of the P-frame encoded data in the bitstream information is determined by the time sequence of the P-frames in the video data. For example, video data may include 100 image frames. The encoding requirement is that there must be at least 28 P-frames between two I-frames. The bitstream information includes the encoded data of the first I-frame, the encoded data of the first P-frame, the encoded data of the second P-frame, ..., the encoded data of the 28th P-frame, the encoded data of the second I-frame, the encoded data of the 29th P-frame, the encoded data of the 30th P-frame, ..., the encoded data of the 56th P-frame, the encoded data of the third I-frame, ... In this case, the encoded data of the first I-frame is obtained by encoding the first image frame in the video data, the encoded data of the first P-frame is obtained by encoding the second image frame in the video data, the encoded data of the second I-frame is obtained by encoding the 30th image frame in the video data, and so on, so that the encoded data obtained by encoding the 100th image frame can be obtained. Thus, the encoded data corresponding to the 100 image frames constitute the bitstream information.

[0059] P-frame encoded data may include motion vectors and residuals. Motion vectors are used to indicate the displacement of an image block between two adjacent image frames and can be two-dimensional vectors. Residuals are used to indicate the pixel differences between the real image and the predicted image. The residuals can show that while the image block is moving, factors such as lighting, shadows, object deformation, and occlusion cause changes in the pixel values ​​of the pixels.

[0060] Step 105: Send the warning data to the electronic device so that the electronic device can extract the features of the warning data, generate warning features, generate second risk information based on the warning features through the second model, and send the second risk information to the vehicle; wherein, the warning data includes code stream data, and the accuracy performance of the first model is lower than that of the second model.

[0061] In this embodiment, the electronic device includes, but is not limited to, chips, servers, and external devices. For example, the electronic device includes an external device, which is independent of the vehicle's main chip. The vehicle's high-speed serial interface connects to the external device, enabling the vehicle to quickly send warning data to the electronic device. The external device may include automotive-grade artificial intelligence (AI) accelerators, AIBOXs, etc., thus adding an external device for regenerating risk information outside the vehicle's main chip. This does not affect the main chip's ability to perform the vehicle's core functions and avoids issues such as cloud transmission latency and the inability to guarantee data security. Both the AI ​​accelerator and the AIBOX are hardware.

[0062] When there are multiple sensor data points, these include video data and second sensor data, where second sensor data refers to sensor data other than video data. The vehicle can encode the video data and compress the second sensor data; for example, compressing the second sensor data to obtain compressed data reduces the amount of data transmitted. Warning data can also include encrypted bitstream data and encrypted compressed data, thereby improving data transmission security.

[0063] The vehicle can wake up the electronic device by sending warning data to it, or the vehicle can send a wake-up signal to the electronic device before sending the warning data. This allows the electronic device to go into sleep mode when it does not need to process the warning data, reducing the energy consumption of the electronic device. When the warning data needs to be processed, it is woken up in response to the warning data or wake-up signal sent by the vehicle, so that the warning data can be processed in a timely manner.

[0064] The electronic device can run a second model, which may include a large model that can obtain second risk information by utilizing more computing and storage resources, thereby improving the accuracy of the generated risk information; and the warning features include bitstream features, which include features extracted from bitstream data, so that the electronic device does not need to decode the encoded bitstream data, thereby improving the processing speed of warning data, improving the real-time performance of generating second risk information, and reducing latency.

[0065] Accuracy performance is used to indicate the reliability and accuracy of the results obtained based on the model. The reliability and accuracy of the results output by the first model are both lower than those output by the second model. The real-time performance of the results output by the first model is greater than that of the results output by the second model. Thus, the vehicle can quickly obtain the first risk level through the first model and determine whether to wake up the electronic devices based on the first risk level, so that the electronic devices can sleep during idle periods, reducing the energy consumption of the electronic devices. In addition, the first model operated by the vehicle is a small model, which also reduces the energy consumption of the vehicle. When the first risk level is high, the vehicle can wake up the electronic devices in time, so that the electronic devices can obtain more accurate second risk information based on the second model.

[0066] Furthermore, the vehicle can determine the processing granularity of video data based on the first risk level, and can control the accuracy of the features extracted by the electronic device through the processing granularity. Thus, the higher the first risk level, the finer the processing granularity. When the first risk level output by the vehicle is higher, the recognition accuracy of the warning data by the electronic device is higher, which improves the reliability and accuracy of the results output by the second model.

[0067] In this embodiment of the disclosure, through the architecture of the vehicle running a first model and the electronic device running a second model, the vehicle can achieve risk detection of the vehicle's environment through a lightweight operating model, enabling the vehicle to continuously identify the risks of the environment for a long time, thereby reducing the vehicle's energy consumption; and through the electronic device, the reliability and accuracy of the risk detection of the vehicle's environment can also be guaranteed. The system employs a mechanism that uses a first model for continuous risk identification and a second model for on-demand wake-up, enabling electronic devices to remain in a sleep state when idle and in a wake-up state when awakened. This reduces the energy consumption of electronic devices and balances detection accuracy and power consumption. Furthermore, electronic devices can extract features without decoding the bitstream data, thereby accelerating the risk identification rate; vehicles can dynamically adjust the processing granularity through the first risk level, and then adjust the identification accuracy of the second model through the processing granularity, thereby significantly reducing the amount of computation and processing latency while ensuring the minimum identification accuracy of the second model, and improving the real-time performance of risk identification of the vehicle's environment.

[0068] In some embodiments, step 102 includes steps 1021 to 1024, wherein: Step 1021: Generate sensor features for each sensor data; wherein, video data is used to indicate the target object; Step 1022: Fuse features from multiple sensors to generate fused features; Step 1023: Generate recognition results based on fused features; wherein, the recognition results are used to indicate the distance between the target object and the vehicle, and the distance is used to indicate the recognition accuracy; Step 1024: Based on the identification results, generate the first risk information.

[0069] In this embodiment of the disclosure, when the vehicle collects data from multiple sensors, it can collect the characteristics of each sensor data to obtain sensor features.

[0070] For example, the vehicle includes four surround-view cameras, millimeter-wave radar, ultrasonic radar, and vibration sensors. The surround-view cameras are used to capture image frames around the vehicle body, and the four surround-view cameras can collect 360° panoramic data around the vehicle. The millimeter-wave radar and ultrasonic radar are both used to detect the distance of objects outside the vehicle and the vehicle itself. In this case, the sensor data includes four video feeds, first radar data acquired by the millimeter-wave radar, second radar data acquired by the ultrasonic radar, and vibration data acquired by the vibration sensors.

[0071] The vehicle can extract features from four videos separately, obtaining video features corresponding to each video; it can also extract features from the first and second radar data separately, obtaining two radar features; and it can extract features from the vibration data, obtaining vibration features. Specifically, when extracting features from the videos, features can be extracted frame-by-frame to obtain image frame features; these image frame features can then be fused to obtain video-level video features, or video frame-level video features can be obtained based on the image frame features, thus obtaining four video features. Finally, the four video features, two radar features, and vibration features can be fused according to time to obtain fused features.

[0072] By fusing features from multiple sensors, the ability to perceive targets under complex lighting conditions, adverse weather conditions, and occlusion conditions can be effectively improved, thereby enhancing the accuracy of the first risk information and the robustness of the first model.

[0073] Step 1023 may include: the vehicle determining the name and identifier of the target object and the distance value between the target object and the vehicle based on the fusion features; determining the recognition accuracy corresponding to the target object based on the correspondence between the distance value and the recognition accuracy; determining the sub-recognition result of the target object based on the recognition accuracy; and generating the recognition result based on the sub-recognition result of the target object.

[0074] The identification results may include the distance between the target object and the vehicle, and may also include at least one of the following: the target object's trajectory, the target object's name, the target object's speed, the target object's texture, size, color, etc. For example, the target object may include other vehicles, pedestrians, animals, bicycles, etc., where "other vehicles" refers to vehicles other than the current vehicle, and the current vehicle may be simply referred to as "vehicle." The vehicle can identify the target objects, including pedestrians, a first vehicle, and a second vehicle, through video data. The distance between the pedestrian and the current vehicle is 2 meters, the distance between the first vehicle and the current vehicle is 3 meters, and the distance between the second vehicle and the current vehicle is 4 meters. The correspondence between distance value and recognition accuracy indicates that when the distance value is less than 3 meters, only the name of the target object is recognized; when the distance value is greater than 3 meters, the movement trajectory, speed, and color of the target object are recognized. The sub-recognition results for pedestrians include the distance value between the pedestrian and the vehicle, the name identifier, the pedestrian's movement trajectory, movement speed, and the color of the coat. The sub-recognition results for the first vehicle include the distance value between the first vehicle and the vehicle itself, the name identifier, the first vehicle's movement trajectory, movement speed, and the color of the paint. The sub-recognition results for the second vehicle include the distance value between the second vehicle and the vehicle itself and the name identifier. This allows the accuracy of target object recognition to be determined by the distance between the target object and the vehicle, reducing unnecessary recognition of distant target objects and improving the speed of generating recognition results.

[0075] The vehicle can analyze the recognition results to obtain initial risk information. For example, it can obtain initial risk information by analyzing the distance between each target object and the vehicle in the recognition results, and whether the vehicle vibrates when a target object passes by.

[0076] For example, in a scenario where a pedestrian passes a vehicle, if the recognition result indicates that the vehicle vibrates when the pedestrian passes by, then the first risk level can be considered high risk; if the recognition result indicates that the minimum distance between the pedestrian and the vehicle is 0.2 meters, then the first risk level can be considered medium risk; if the recognition result indicates that the minimum distance between the pedestrian and the vehicle is 2 meters, then the first risk level can be considered low risk.

[0077] In some embodiments, the method further includes steps 1061 to 1063, wherein: Step 1061: Receive second risk information sent by an electronic device; wherein the second risk information is used to indicate a second risk level; Step 1062: When the second risk level is the first preset level, in response to the vehicle power-on command, generate the first prompt information based on the second risk information; Step 1063: If the second risk level is the second preset level, generate a second prompt message based on the second risk information and send the second prompt message to the user equipment; The second preset level is greater than the first preset level.

[0078] In this embodiment of the present disclosure, the vehicle is able to receive second risk information sent by an electronic device. The second risk information includes at least one of the following: a second risk level, a second risk scenario type, the time of the risk occurrence, a target object that may cause vehicle damage, and the cause and effect of the event, wherein the second risk scenario type may correspond to the second risk level.

[0079] The vehicle can determine whether to immediately push second risk information to the user's device based on the second risk level. If the risk level is low, a first alert will be output based on the second risk information when the user starts the vehicle, allowing the user to determine whether the vehicle is damaged. If the risk level is high, a second alert will be output immediately based on the second risk information, enabling the user to be promptly informed of potential vehicle damage.

[0080] The prompt information may include a first prompt or a second prompt, and may include video data, as well as second risk information, so that users can directly see the possible cause of vehicle damage based on video data without having to specifically search for the video of the time period that caused the vehicle damage from the stored video.

[0081] The initial notification can be provided through at least one of the vehicle lights, speakers, or display screen. For example, when the vehicle is first powered on, the vehicle can automatically control the display screen to show video data.

[0082] In some embodiments, the method further includes steps 1071 to 1072, wherein: Step 1071: If the storage conditions are not met at the first risk level, or if the storage conditions are not met at the second risk level, determine the storage area corresponding to the video data as the first storage area. Step 1072: If the storage conditions are met at the second risk level, determine the second storage area corresponding to the video data; The first storage duration corresponding to the first storage area is less than the second storage duration corresponding to the second storage area.

[0083] In this embodiment of the present disclosure, the vehicle may include a memory, and the memory may be partitioned into storage areas, wherein a first storage area is used to indicate a cyclic storage area; and a second storage area is used to indicate a permanent storage area or a long-term storage area, etc.

[0084] The circular storage area is an area that can be used automatically in a cyclical manner. For example, if the circular storage area is used to store video data within 24 hours, then if the first storage area has already stored video data within 24 hours, the latest video data can be used to overwrite the already stored video data, thereby enabling the first storage area to be used to store video data in a cyclical manner.

[0085] The second storage area may include a region for permanently storing video data. After video data is stored in this permanent storage area, the vehicle will not automatically delete the video data stored there. This avoids the problem of video data being overwritten by the latest video data after vehicle damage, making it impossible to view videos from the period of vehicle damage. The second storage area may also include a region for long-term storage of video data. After video data is stored in this long-term storage area, the vehicle can store the video data for an extended period, thus preventing the video data stored in the permanent storage area from occupying storage space.

[0086] Storage conditions can be used to indicate that when the risk level is less than a first preset level, the storage area corresponding to the video data is a first storage area; when the risk level is equal to the first preset level, the storage area corresponding to the video data is a long-term storage area; and when the risk level is equal to a second preset level, the storage area corresponding to the video data is a permanent storage area. The risk level includes either a first risk level or a second risk level.

[0087] The vehicle can respond to user-inputted add / delete commands and update the video data stored in the second storage area, thereby automatically adding or deleting video areas. For example, video data stored in a long-term storage area can be transferred to a permanent storage area.

[0088] If the duration of unattended third-party video data stored in the long-term storage area is longer than the storage duration of the long-term storage area, the vehicle can store the third-party video data in the permanent storage area. For example, if the storage duration of the long-term storage area is 15 days, and the third-party video data is not viewed by the user within 15 days, the vehicle can automatically change the storage area corresponding to the third-party video data to the permanent storage area, thereby preventing the third-party video data from being overwritten due to untimely viewing by the user.

[0089] Figure 2 A flowchart of another risk identification method provided in the embodiments of this application is shown below. Figure 2 As shown, the method includes: Step 201: Receive warning data sent by the vehicle; wherein the warning data includes bitstream data, which includes data from at least one sensor acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; generate first risk information based on the at least one sensor data using a first model; if the first risk level indicated by the first risk information meets the first risk condition, determine the processing granularity based on the first risk level; generate data by encoding the video data based on the processing granularity. Step 202: Extract features from the warning data to generate warning features; wherein, the warning features include bitstream features, which are features extracted from bitstream data; Step 203: Using the second model, generate second risk information based on the early warning features; wherein, the accuracy performance of the first model is lower than that of the second model. Step 204: Send the second risk information to the vehicle.

[0090] In this embodiment of the disclosure, the warning data may include bitstream data, and may also include other sensor data, or compressed data corresponding to other sensor data. During the feature extraction process of the warning data, there is no need to decode the bitstream data; the bitstream features are directly extracted from the bitstream data. This reduces the time required to decode the bitstream data and the time required to generate warning features, thereby accelerating the generation rate of the second risk information, which is then sent to the vehicle.

[0091] In some embodiments, the video data includes keyframes and forward prediction frames, the bitstream data includes first coded data corresponding to the keyframes and second coded data corresponding to the forward prediction frames, the second coded data includes motion vector data and residual data, and the warning features include coded features; step 202 includes steps 2021 to 2024, wherein: Step 2021: Extract spatial features from the first encoded data to generate spatial features; Step 2022: Extract features from motion vector data to generate motion features, and extract features from residual data to generate residual features; Step 2023: Concatenate the motion features and residual features to generate concatenated features; Step 2024: Combine the spatial features and the splicing features to generate coded features.

[0092] In this embodiment of the disclosure, the first encoded data can also be referred to as I-frame encoded data, and the second encoded data can also be referred to as P-frame encoded data. Keyframes and forward prediction frames can be collectively referred to as image frames or video frames. The video data includes keyframes and forward prediction frames arranged in temporal order. When stitching together the features of multiple image frames, spatial features and stitching features can be stitched together based on the temporal order of each image frame in the video data to generate encoded features. The encoded features include features corresponding to the first encoded data and features corresponding to the second encoded data arranged in temporal order. The encoded features can be referred to as a temporal feature sequence.

[0093] The bitstream data can be referred to as the encoded video stream, that is, the encoded video stream. The bitstream data includes I-frame encoded data and P-frame encoded data arranged in time sequence. Figure 3 A flowchart for feature extraction provided in an embodiment of this application is shown below. Figure 3 As shown, it includes: Step 31: Obtain the encoded video stream; Step 32: Analyze the encoded video stream to obtain I-frame encoded data and P-frame encoded data arranged in time sequence; Step 33: Extract the spatial features of each I-frame encoded data. Step 34: For each P-frame encoded data, extract the motion vector and residual from the P-frame encoded data; Step 35: Extract the motion features of the motion vectors and the residual features of the residuals; Step 36: For each P-frame encoded data, the motion features and residual features corresponding to the same P-frame encoded data are concatenated to obtain the concatenated features; Step 37: According to the temporal sequence of I-frame encoded data and P-frame encoded data in the bitstream data, concatenate the spatial features corresponding to I-frame encoded data and the concatenation features corresponding to P-frame encoded data to obtain a temporal feature sequence. Step 38: Input the time series feature sequence into the large model.

[0094] In this embodiment of the disclosure, such as Figure 3 As shown, the electronic device analyzes the bitstream data and separates each I-frame encoded data and each P-frame encoded data. For example, the bitstream data may include the frame headers corresponding to each encoded data. By locating the position of the frame headers, the start position of the I-frame encoded data and the start position of the P-frame encoded data can be determined, thereby separating each I-frame encoded data and each P-frame encoded data.

[0095] The second model comprises a larger model. Spatial structure features are extracted from I-frame encoded data as the image baseline; motion vectors and residuals are extracted from P-frame encoded data to further extract motion and detail features. These features are then aligned and concatenated temporally to form a continuous temporal feature sequence. This temporal feature sequence is directly input into the larger model for understanding and reasoning to obtain the second risk information.

[0096] In some embodiments, the warning data further includes first sensor data; step 202 further includes steps 2025 to 2026, wherein: Step 2025: Extract features from the first sensor data to generate the first feature; Step 2026: Fuse the first feature and the encoded feature to generate the warning feature.

[0097] In this embodiment of the disclosure, when the warning data only includes bitstream data, the second model includes a model that processes the coding features; when the warning data includes bitstream data and first sensor data, the second model may include a model that processes multiple modes.

[0098] The first sensor data may include the aforementioned vibration data and / or radar data, etc. For example, the first sensor data may include the aforementioned vibration data and radar data. The electronic device may extract features from the vibration data and radar data respectively to obtain vibration features and radar features. Then, the vibration features, radar features, and coded features are fused to generate early warning features.

[0099] In some embodiments, the second risk information is used to indicate a second risk level; step 204 includes: if the second risk level meets a second risk condition, sending the second risk information to the vehicle, and / or generating a third alert based on the second risk information and sending the third alert to the user equipment. The second risk condition is represented by step 2041 or step 2042, wherein: Step 2041: If the second risk level is less than or equal to the second preset level, send the second risk information to the vehicle; or, Step 2042: If the second risk level is less than or equal to the first preset level, send the second risk information to the vehicle; if the second risk level is the second preset level, send the second risk information to the vehicle, and send the third prompt information to the user device; wherein the second preset level is greater than the first preset level.

[0100] In this embodiment of the disclosure, the preset level may include a first preset level, a second preset level, and a third preset level, wherein the third preset level is lower than the first preset level, the first preset level is lower than the second preset level, and the third preset level may include a low level, the first preset level may include a medium level, and the second preset level may include a high level.

[0101] The second risk condition can be used to indicate that, regardless of whether the second risk level is at any level, the second risk information will be sent to the vehicle without reminding the user. That is, in step 2041, the second risk information will be sent to the vehicle when the second risk level is at the first preset level, the second preset level, or the third preset level.

[0102] Alternatively, the second risk condition can be used to instruct that when the second risk level indicates a high risk of vehicle damage, the user should be notified immediately; when the second risk level indicates a low risk of vehicle damage, the second risk information should be sent to the vehicle, and the vehicle should then perform subsequent steps. That is, in step 2042, when the second risk level is either the first preset level or the third preset level, the second risk information will be sent to the vehicle; when the second risk level is the second preset level, in addition to sending the second risk information to the vehicle, the user should also be promptly reminded so that the user can decide whether to check the vehicle immediately.

[0103] In some embodiments, the method further includes: Based on the idle time being greater than the duration threshold, the electronic device is determined to be in sleep mode. Idle time is used to indicate when an electronic device is idle.

[0104] In this embodiment of the disclosure, the moment when the electronic device sends the second risk information can be used as the start time, and the moment obtained by adding a duration threshold to the start time can be used as the end time. The duration from the start time to the end time is equal to the duration threshold.

[0105] If the electronic device does not receive the next warning data during the period from the start time to the end time, it is determined that the idle time is longer than the time threshold, and the electronic device can enter a sleep state, thereby reducing the energy consumption of the electronic device.

[0106] Figure 4 A schematic diagram of a risk identification system provided in this application embodiment is shown below. Figure 4 As shown, it includes: Vehicle 401 is configured to acquire at least one sensor data when the risk identification mode is activated, wherein the at least one sensor data includes video data; generate first risk information based on the at least one sensor data using a first model, wherein the first risk information indicates a first risk level; determine processing granularity based on the first risk level if the first risk level meets a first risk condition; encode the video data based on the processing granularity to generate bitstream data; and send warning data to electronic device 402, wherein the warning data includes bitstream data. Electronic device 402 is used to receive warning data sent by vehicle 401; extract features from the warning data to generate warning features; wherein the warning features include code stream features, which are features extracted from code stream data; generate second risk information based on the warning features through a second model; wherein the accuracy performance of the first model is lower than that of the second model; and send the second risk information to vehicle 401.

[0107] In some embodiments, vehicle 401 is used for: For each sensor data point, sensor features are generated; video data is used to indicate the target object. Multiple sensor features are fused to generate a fused feature; Based on the fusion features, a recognition result is generated; the recognition result is used to indicate the distance between the target object and the vehicle, and the distance is used to indicate the recognition accuracy. Based on the identification results, the first risk information is generated.

[0108] In some embodiments, the video data includes keyframes and forward prediction frames, the bitstream data includes first coded data corresponding to the keyframes and second coded data corresponding to the forward prediction frames, the second coded data includes motion vector data and residual data, and the warning features include coded features; the electronic device 402 is used for: Spatial features are extracted from the first encoded data to generate spatial features; Features are extracted from motion vector data to generate motion features, and features are extracted from residual data to generate residual features; The motion features and residual features are spliced ​​together to generate spliced ​​features; Spatial features and splicing features are spliced ​​together to generate coded features.

[0109] In some embodiments, the warning data further includes first sensor data; electronic device 402, for: Features are extracted from the data from the first sensor to generate the first feature; The first feature and the encoded feature are fused to generate the warning feature.

[0110] In some embodiments, the second risk information is used to indicate a second risk level; the electronic device 402 is configured to: send the second risk information to the vehicle if the second risk level is less than or equal to a second preset level; or... If the second risk level is less than or equal to the first preset level, the second risk information is sent to vehicle 401; if the second risk level is the second preset level, the second risk information is sent to vehicle 401, and the third prompt information is sent to the user device. The second preset level is greater than the first preset level.

[0111] In some embodiments, electronic device 402 is configured to: determine that the electronic device is in a sleep state based on the idle time being greater than a duration threshold. Idle time is used to indicate when an electronic device is idle.

[0112] In some embodiments, vehicle 401 is configured to receive second risk information sent by electronic device 402; wherein the second risk information is used to indicate a second risk level; When the second risk level is the first preset level, in response to the vehicle power-on command, a first prompt message is generated based on the second risk information; When the second risk level is the second preset level, a second prompt message is generated based on the second risk information and sent to the user device. The second preset level is greater than the first preset level.

[0113] In some embodiments, vehicle 401 is configured to determine the storage area corresponding to the video data as the first storage area when the storage conditions are not met at the first risk level, or when the storage conditions are not met at the second risk level. If the storage conditions are met at the second risk level, determine the second storage area corresponding to the video data; The first storage duration corresponding to the first storage area is less than the second storage duration corresponding to the second storage area.

[0114] In some embodiments, the vehicle includes a main chip, and the electronic devices include an AIBOX; the first model includes a small model, and the second model includes a large model. Figure 5 A signaling diagram for risk identification provided in an embodiment of this application, such as Figure 5 As shown, it includes: Step 51: The main chip detects that the sentry mode is enabled; Step 52: The main chip processes the acquired data from at least one sensor using a small model to obtain the first risk information, thereby achieving low-power continuous monitoring; the data from at least one sensor includes the video corresponding to the first time period. Step 53: When the first risk level indicated by the first risk information is high, the main chip wakes up AIBOX; Step 54: The main chip determines the processing granularity based on the first risk level, and encodes the video corresponding to the second time period based on the processing granularity to obtain the bitstream; the bitstream is sent to AIBOX; the second time period includes the first time period and the time periods before and after the first time period; Step 55: When AIBOX switches from dormant to awake state, it uses a large model to perform compressed domain inference based on the bitstream to obtain the second risk information. Step 56: AIBOX sends the second risk information to the main chip; Step 57: The main chip performs graded processing in the storage dimension and the prompt dimension respectively, based on the second risk level indicated by the second risk information; Step 58: AIBOX calculates the idle time and determines whether the idle time is greater than the time threshold. If yes, proceed to step 55; otherwise, proceed to step 59. Step 59: AIBOX switches its state from wake-up to sleep.

[0115] In this embodiment of the disclosure, the compression domain refers to the domain in which data is compressed. Compression domain inference is used to indicate how to obtain second risk information through the second model without decompressing the encoded bitstream data.

[0116] Tiered processing refers to determining the storage area corresponding to video data based on different risk levels, as well as determining whether to remind the user, and if so, whether to remind the user immediately.

[0117] exist Figure 5 In the illustrated process, the second model may include a model for performing compressed domain inference, thereby enabling the extraction and analysis of features in the bitstream without decoding the bitstream, thus accelerating the generation of second risk information.

[0118] In some embodiments, the vehicle includes a main chip, and the electronic devices include an AIBOX. Figure 6 This is a schematic diagram of a risk identification architecture provided in an embodiment of this application, such as... Figure 6As shown, the risk identification system has a multi-layered sensor perception layer, a vehicle-side computing layer, and an AIBOX large model layer. Specifically, the risk identification system includes sensors, a main chip, and an AIBOX, which can be integrated into the same vehicle. The multi-layered sensor layer corresponds to multiple sensors, including cameras, millimeter-wave radar, ultrasonic radar, and vibration sensors. The vehicle-side computing layer corresponds to the main chip, which includes a memory, a small model running module, a control module, an encoding module, and an alarm module. The AIBOX large model layer corresponds to the AIBOX, which includes a communication module, a power management module, a compressed domain feature extraction module, a large model running module, and an alarm module.

[0119] In this embodiment, the main chip and AIBOX can be integrated into the same vehicle, thereby avoiding the problems of reduced data security and increased transmission latency caused by the interaction between the main chip and the cloud; or, the AIBOX can be pluggable hardware, so that users can control whether the risk identification mode is enabled by plugging and unplugging the AIBOX.

[0120] The main chip may include an alarm module for controlling the display screen to show the user a video and / or second risk information corresponding to a first preset level when the user starts the vehicle; alternatively, it may send a prompt message corresponding to a second preset level to the user equipment. The AIBOX may include an alarm module for sending a prompt message corresponding to a second preset level to the user equipment.

[0121] The descriptions of the system embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the system provided by this disclosure can be used to execute the methods described in the method embodiments above. For technical details not disclosed in the system embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0122] Figure 7 This is a schematic diagram of the structure of a risk identification device provided in an embodiment of this application, as shown below. Figure 7 As shown, the risk identification device 700 includes: The acquisition module 701 is used to acquire at least one sensor data when the risk identification mode is enabled; wherein the at least one sensor data includes video data. The first generation module 702 is used to generate first risk information based on at least one sensor data through a first model; wherein the first risk information is used to indicate a first risk level. The first determining module 703 is used to determine the processing granularity based on the first risk level when the first risk level meets the first risk condition. The second generation module 704 is used to encode video data based on processing granularity to generate bitstream data; The first sending module 705 is used to send the warning data to the electronic device so that the electronic device can extract the features of the warning data, generate warning features, generate second risk information based on the warning features through the second model, and send the second risk information to the vehicle. The early warning data includes code stream data, and the accuracy performance of the first model is lower than that of the second model.

[0123] In some embodiments, the first generation module 702 is configured to: For each sensor data point, sensor features are generated; video data is used to indicate the target object. Multiple sensor features are fused to generate a fused feature; Based on the fusion features, a recognition result is generated; the recognition result is used to indicate the distance between the target object and the vehicle, and the distance is used to indicate the recognition accuracy. Based on the identification results, the first risk information is generated.

[0124] In some embodiments, the first transmitting module 705 is configured to: Receive second risk information sent by an electronic device; wherein the second risk information is used to indicate a second risk level; When the second risk level is the first preset level, in response to the vehicle power-on command, a first prompt message is generated based on the second risk information; When the second risk level is the second preset level, a second prompt message is generated based on the second risk information and sent to the user device. The second preset level is greater than the first preset level.

[0125] In some embodiments, the risk identification device 700 further includes: a second determining module, used for If the storage conditions are not met at the first risk level, or if the storage conditions are not met at the second risk level, the storage area corresponding to the video data shall be designated as the first storage area. If the storage conditions are met at the second risk level, determine the second storage area corresponding to the video data; The first storage duration corresponding to the first storage area is less than the second storage duration corresponding to the second storage area.

[0126] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0127] Figure 8 This is a schematic diagram of another risk identification device provided in an embodiment of this application, as shown below. Figure 8 As shown, the risk identification device 800 includes: The receiving module 801 is used to receive warning data sent by the vehicle; wherein the warning data includes bitstream data, and the bitstream data includes data from at least one sensor acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; generating first risk information based on the at least one sensor data through a first model; determining the processing granularity based on the first risk level indicated by the first risk information if the first risk level meets the first risk condition; and generating data by encoding the video data based on the processing granularity. The third generation module 802 is used to extract features from the warning data and generate warning features; wherein, the warning features include code stream features, which are features extracted from the code stream data; The fourth generation module 803 is used to generate second risk information based on early warning features using the second model; wherein the accuracy performance of the first model is lower than that of the second model. The second sending module 804 is used to send the second risk information to the vehicle.

[0128] In some embodiments, the video data includes keyframes and forward prediction frames, the bitstream data includes first coded data corresponding to the keyframes and second coded data corresponding to the forward prediction frames, the second coded data includes motion vector data and residual data, and the warning features include coded features; the third generation module 802 is used for: Spatial features are extracted from the first encoded data to generate spatial features; Features are extracted from motion vector data to generate motion features, and features are extracted from residual data to generate residual features; The motion features and residual features are spliced ​​together to generate spliced ​​features; Spatial features and splicing features are spliced ​​together to generate coded features.

[0129] In some embodiments, the warning data further includes first sensor data; the third generation module 802 is configured to: Features are extracted from the data from the first sensor to generate the first feature; The first feature and the encoded feature are fused to generate the warning feature.

[0130] In some embodiments, the second risk information is used to indicate a second risk level; the second sending module 804 is used to: If the second risk level is less than or equal to the second preset level, the second risk information will be sent to the vehicle; or, If the second risk level is less than or equal to the first preset level, the second risk information is sent to the vehicle; if the second risk level is the second preset level, the second risk information is sent to the vehicle; and, the third prompt information is sent to the user device. The second preset level is greater than the first preset level.

[0131] In some embodiments, the risk identification device 800 further includes: Based on the idle time being greater than the duration threshold, the electronic device is determined to be in sleep mode. Idle time is used to indicate when an electronic device is idle.

[0132] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0133] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0134] Figure 9 This is a schematic diagram of the structural composition of a vehicle 900 provided in an embodiment of this application, as shown below. Figure 9As shown, vehicle 900 includes: processor 901, communication interface 902, and memory 903, wherein: The processor 901 typically controls the overall operation of the vehicle 900, which may be to implement the risk identification method provided in the embodiments of this application.

[0135] The communication interface 902 enables the vehicle 900 to communicate with other terminals or servers via a network.

[0136] The memory 903 is configured to store instructions and applications executable by the processor 901, and can also cache data to be processed or already processed by the processor 901 and various modules in the vehicle 900 (e.g., image data, audio data, voice communication data, and video communication data). It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 901, the communication interface 902, and the memory 903 can be performed via bus 904.

[0137] Figure 10 This is a schematic diagram of the composition structure of an electronic device 1000 provided in an embodiment of this application, as shown below. Figure 10 As shown, the device includes: a processor 1001, a communication interface 1002, and a memory 1003, wherein: The processor 1001 typically controls the overall operation of the electronic device 1000, which may be to implement the risk identification method provided in the embodiments of this application.

[0138] The communication interface 1002 enables the electronic device 1000 to communicate with other terminals or servers via a network.

[0139] The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1001 and various modules in the electronic device 1000. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 1001, the communication interface 1002, and the memory 1003 can be performed via bus 1004.

[0140] This application provides a computer program product or computer program that includes computer instructions stored in a readable storage medium. The processor of an electronic device reads the computer instructions from the readable storage medium and executes the computer instructions, causing the electronic device to perform the risk identification method described in this application.

[0141] This application provides a readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, will cause the processor to execute the risk identification method provided in this application.

[0142] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0143] The aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that other electronic devices can also implement the functions of the aforementioned processor, and this application does not specifically limit the specific implementation.

[0144] The aforementioned computer storage media / memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0146] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0147] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0148] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0149] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0150] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A risk identification method, characterized in that, The method includes: When the risk identification mode is enabled, at least one sensor data is acquired; wherein, the at least one sensor data includes video data; Based on the data from the at least one sensor, a first risk information is generated using a first model; wherein the first risk information is used to indicate a first risk level. If the first risk level meets the first risk condition, the processing granularity is determined based on the first risk level. Based on the processing granularity, the video data is encoded to generate bitstream data; The warning data is sent to an electronic device, which extracts features from the warning data to generate warning features. Based on the warning features, a second model is used to generate second risk information, which is then sent to the vehicle. The warning data includes the code stream data, and the accuracy performance of the first model is lower than that of the second model.

2. The method according to claim 1, characterized in that, The step of generating first risk information based on the at least one sensor data using a first model includes: For each of the sensor data, a sensor feature is generated; wherein the video data is used to indicate the target object; The features from multiple sensors are fused to generate a fused feature; Based on the fusion features, a recognition result is generated; wherein the recognition result is used to indicate the distance between the target object and the vehicle, and the distance is used to indicate the recognition accuracy; Based on the identification results, the first risk information is generated.

3. The method according to claim 1, characterized in that, The method further includes: Receive second risk information sent by the electronic device; wherein the second risk information is used to indicate a second risk level; When the second risk level is the first preset level, in response to the vehicle power-on command, a first prompt message is generated based on the second risk information; When the second risk level is the second preset level, a second prompt message is generated based on the second risk information and sent to the user device; The second preset level is greater than the first preset level.

4. The method according to claim 3, characterized in that, The method further includes: If the storage conditions are not met at the first risk level, or if the storage conditions are not met at the second risk level, the storage area corresponding to the video data is determined as the first storage area. If the second risk level meets the storage conditions, determine the second storage area corresponding to the video data; Wherein, the first storage duration corresponding to the first storage area is less than the second storage duration corresponding to the second storage area.

5. A risk identification method, characterized in that, The method includes: The system receives warning data sent by a vehicle; wherein the warning data includes bitstream data, which includes data from at least one sensor acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; a first risk information is generated based on the at least one sensor data using a first model; if the first risk level indicated by the first risk information meets a first risk condition, a processing granularity is determined based on the first risk level; and data is generated after encoding the video data based on the processing granularity. Features are extracted from the warning data to generate warning features; wherein, the warning features include bitstream features, which are features extracted from the bitstream data; Based on the warning features, a second risk information is generated using a second model; wherein the accuracy performance of the first model is lower than that of the second model. The second risk information is sent to the vehicle.

6. The method according to claim 5, characterized in that, The video data includes keyframes and forward prediction frames, the bitstream data includes first coded data corresponding to the keyframes and second coded data corresponding to the forward prediction frames, the second coded data includes motion vector data and residual data, and the warning feature includes coded features; The step of extracting features from the early warning data to generate early warning features includes: Spatial features are extracted from the first encoded data to generate spatial features; Features are extracted from the motion vector data to generate motion features, and features are extracted from the residual data to generate residual features; The motion features and the residual features are concatenated to generate concatenated features; The spatial features and the splicing features are spliced ​​together to generate the encoded features.

7. The method according to claim 6, characterized in that, The early warning data also includes first sensor data; the extraction of features from the early warning data to generate early warning features includes: The features of the first sensor data are extracted to generate the first feature; The first feature and the encoded feature are fused to generate the warning feature.

8. The method according to claim 5, characterized in that, The second risk information is used to indicate the second risk level; Sending the second risk information to the vehicle includes: If the second risk level is less than or equal to the second preset level, the second risk information will be sent to the vehicle; or, If the second risk level is less than or equal to the first preset level, the second risk information is sent to the vehicle; if the second risk level is the second preset level, the second risk information is sent to the vehicle, and the third prompt information is sent to the user device. The second preset level is greater than the first preset level.

9. The method according to any one of claims 5 to 8, characterized in that, The method further includes: Based on the idle time being greater than the duration threshold, the electronic device is determined to be in sleep mode. The idle time is used to indicate that the electronic device is idle.

10. A risk identification system, characterized in that, include: A vehicle is configured to acquire at least one sensor data when a risk identification mode is activated; wherein the at least one sensor data includes video data; generate first risk information based on the at least one sensor data using a first model; wherein the first risk information indicates a first risk level; determine a processing granularity based on the first risk level when the first risk level meets a first risk condition; encode the video data based on the processing granularity to generate a bitstream data; and send warning data to an electronic device; wherein the warning data includes the bitstream data. The electronic device is used to receive warning data sent by a vehicle; extract features from the warning data to generate warning features; wherein the warning features include bitstream features, which are features extracted from the bitstream data; generate second risk information based on the warning features using a second model; wherein the accuracy performance of the first model is lower than that of the second model; and send the second risk information to the vehicle.

11. A risk identification device, characterized in that, The device includes: The acquisition module is used to acquire at least one sensor data when the risk identification mode is enabled; wherein, the at least one sensor data includes video data. A first generation module is configured to generate first risk information based on the data from the at least one sensor using a first model; wherein the first risk information is used to indicate a first risk level. The first determining module is used to determine the processing granularity based on the first risk level, provided that the first risk level meets the first risk condition. The second generation module is used to encode the video data based on the processing granularity to generate bitstream data; The first sending module is used to send the warning data to the electronic device, so that the electronic device can extract the features of the warning data, generate warning features, generate second risk information based on the warning features through the second model, and send the second risk information to the vehicle. The warning data includes the code stream data, and the accuracy performance of the first model is lower than that of the second model.

12. A risk identification device, characterized in that, The device includes: A receiving module is configured to receive warning data sent by a vehicle; wherein the warning data includes bitstream data, the bitstream data including at least one sensor data acquired by the vehicle when the risk recognition mode is enabled; wherein the at least one sensor data includes video data; a first risk information is generated based on the at least one sensor data using a first model; if the first risk level indicated by the first risk information meets a first risk condition, a processing granularity is determined based on the first risk level; and data is generated after encoding the video data based on the processing granularity. The third generation module is used to extract features from the warning data and generate warning features; wherein, the warning features include bitstream features, which are features extracted from the bitstream data; The fourth generation module is used to generate second risk information based on the warning features using the second model; wherein the accuracy performance of the first model is lower than that of the second model. The second sending module is used to send the second risk information to the vehicle.

13. A vehicle, the device comprising a processor and a memory storing instructions executable by the processor; characterized in that, When the instructions are executed by the processor, the method described in any one of claims 1 to 4 is implemented.

14. An electronic device, the device comprising a processor and a memory storing instructions executable by the processor; characterized in that, When the instructions are executed by the processor, the method described in any one of claims 5 to 9 is implemented.

Images