A vehicle-mounted visual perception resource scheduling method and device and a vehicle

Abstract

This invention relates to the field of automotive electronics technology, and proposes a method, device, and vehicle for scheduling in-vehicle visual perception resources. The method includes: receiving multiple video data streams and vehicle status information; performing perception processing on the multiple video data streams to output a primary perception event; calculating a safety urgency score for the current driving scenario based on the primary perception event and the vehicle status information; determining the priority configuration of an analysis task based on the safety urgency score and the scenario type of the current driving scenario; dynamically adjusting the allocation of computing resources for the analysis task based on the priority configuration and the computing resource status; and executing the analysis task based on the adjusted computing resources to obtain accurate perception results. This invention achieves the adaptation of vehicle computing resources to actual perception needs under different driving scenarios.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics technology, and in particular to a method, device, and vehicle for scheduling in-vehicle visual perception resources. Background Technology

[0002] As vehicles become increasingly intelligent, in-vehicle vision perception systems typically need to simultaneously perform multiple perception tasks, such as object detection, lane line recognition, and obstacle recognition, and complete the corresponding data processing through the processing resources of the in-vehicle computing platform. During actual vehicle operation, the driving scenario constantly changes, and the required perception tasks and processing demands vary across different scenarios. For example, in typical driving scenarios, some perception tasks may have lower processing requirements, while in scenarios where objects appear around the vehicle or road conditions change, the relevant perception tasks may require more timely or in-depth data processing. Existing in-vehicle vision perception systems usually execute perception tasks according to pre-set task or resource configurations, making it difficult to adapt computing resources to actual perception needs as the current driving scenario changes. Summary of the Invention

[0003] The purpose of this invention is to provide a method, device and vehicle for scheduling vehicle visual perception resources, so as to achieve the adaptation between the vehicle's computing resources and actual perception needs in different driving scenarios.

[0004] To achieve the above objectives, a first aspect of the present invention provides a method for scheduling vehicle-mounted visual perception resources, comprising: receiving multiple video data streams and vehicle status information; performing perception processing on the multiple video data streams to output a primary perception event; calculating a safety urgency score for the current driving scenario based on the primary perception event and the vehicle status information; dynamically determining the safety urgency score based on the risk level and risk change trend of the current driving scenario; determining a priority configuration for an analysis task based on the safety urgency score and the scenario type of the current driving scenario; dynamically adjusting the allocation of computing resources for the analysis task based on the priority configuration and computing resource status; and executing the analysis task based on the adjusted computing resources to obtain accurate perception results.

[0005] In some embodiments, determining the priority configuration of analysis tasks based on the safety urgency score and the scenario type of the current driving scenario includes: determining a set of advanced analysis tasks corresponding to the current driving scenario based on the scenario type; determining the risk correlation between each advanced analysis task in the set of advanced analysis tasks and the risk factors in the current driving scenario; and determining the execution priority of each advanced analysis task based on the safety urgency score and the risk correlation.

[0006] Specifically, determining the execution priority of each advanced analysis task based on the security urgency score and the risk correlation includes: determining the relative priority relationship between each advanced analysis task based on the risk correlation, and adjusting the relative priority relationship based on the security urgency score; wherein, advanced analysis tasks with a high risk correlation with the current risk factor are given higher execution priority when the security urgency score increases, while advanced analysis tasks with a low risk correlation with the current risk factor are given lower execution priority.

[0007] Specifically, the risk factors are determined based on the vehicle status information and the primary perception events, including at least one of target approach, abnormal target motion state, driver line of sight deviation, driver fatigue, and abnormal occupant posture.

[0008] Specifically, the correspondence between the scenario type and the set of advanced analysis tasks, as well as the risk correlation between the risk factors and the advanced analysis tasks, are all pre-stored in the scenario-task mapping strategy library.

[0009] In some embodiments, the computing resources include a neural network processor (NPU) and a graphics processing unit (GPU) configured in an on-chip system-on-a-chip (SoC). The dynamic adjustment of the computing resource allocation for the analysis task includes: allocating the analysis task to the NPU and / or GPU for execution according to the priority of the analysis task.

[0010] In some embodiments, when a newly activated advanced analysis task has a high execution priority and the currently available computing resources are insufficient to meet the computing needs of the advanced analysis task, resource adjustments are made to the currently executing low-priority analysis task.

[0011] Specifically, resource adjustments for the low-priority analysis task include at least one of reducing the execution frequency of the low-priority analysis task, pausing the low-priority analysis task, and migrating the low-priority analysis task to other computing resources.

[0012] In some embodiments, the precise perception result is obtained by the analysis task further analyzing the primary perception event and is used to accurately identify target objects, risk states, or driving environments in the current driving scenario.

[0013] In some embodiments, multiple precise perception results are fused and analyzed to determine the risk level of the current driving scenario based on the fusion analysis results, and corresponding collaborative control commands are generated based on the risk level to control the vehicle's display devices, alarm devices, and / or body actuators to perform corresponding linkage operations.

[0014] A second aspect of the present invention provides an in-vehicle visual perception resource scheduling device, comprising: a primary perception module configured to receive multiple video data streams and vehicle status information, perform perception processing on the multiple video data streams, and output a primary perception event; a safety assessment module configured to calculate a safety urgency score for the current driving scenario based on the primary perception event and the vehicle status information; a priority configuration module configured to determine a priority configuration for an analysis task based on the safety urgency score and the scenario type of the current driving scenario; and a resource scheduling module configured to dynamically adjust the allocation of computing resources for the analysis task based on the priority configuration and the computing resource status, and execute the analysis task based on the adjusted computing resources to obtain accurate perception results.

[0015] A third aspect of the present invention provides a vehicle including a processor and a memory, the memory storing a computer program that, when executed on the processor, causes the vehicle to perform the on-board visual perception resource scheduling method.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when run on a processor, causes the processor to execute the vehicle-mounted visual perception resource scheduling method.

[0017] Compared with the prior art, the present invention has the following beneficial effects: By receiving multiple video data streams and vehicle status information, and processing the video data to output primary perception events, the vehicle-mounted visual perception system acquires basic perception information of the current driving scenario. Based on this, a safety urgency score is calculated using the primary perception events and vehicle status information, transforming the safety risk level of the current driving scenario into a quantitative basis for subsequent scheduling. Furthermore, the priority configuration of analysis tasks is determined based on the safety urgency score and the scenario type of the current driving scenario, ensuring that the analysis tasks match the risk state of the current driving scenario. Then, the allocation of computing resources for the analysis tasks is dynamically adjusted based on the priority configuration and computing resource status, prioritizing high-priority analysis tasks within the current driving scenario with limited computing resources. The corresponding analysis tasks are then executed based on the adjusted computing resources to obtain accurate perception results. Thus, the correspondence between visual perception tasks and computing resources can be dynamically adjusted according to the risk state of the driving scenario, reducing unnecessary consumption of computing resources in low-risk scenarios and improving resource availability for critical analysis tasks in high-risk scenarios. This improves the perception response capability and resource utilization efficiency of the vehicle-mounted visual perception system under limited computing resource conditions. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0019] Figure 1 This is a schematic diagram of an example environment in which various embodiments of the present invention can be implemented; Figure 2 This is a schematic diagram of the overall process of the vehicle-mounted visual perception resource scheduling method provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the process of determining the priority configuration of analysis tasks according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the vehicle-mounted visual perception resource scheduling device in an embodiment of the present invention. Detailed Implementation

[0020] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application. In the description of the embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment".

[0021] Existing vehicle-mounted visual perception systems typically execute perception tasks according to pre-set task or resource configurations. The computing resources of each perception subsystem are independent and cannot be shared across domains. Furthermore, peak computing power must be allocated independently based on worst-case scenarios, resulting in significant idle computing power and low utilization rates in typical driving scenarios. Simultaneously, in high-risk scenarios such as the appearance of objects around the vehicle or sudden changes in road conditions, the lack of a dynamic scheduling mechanism based on real-time scenario risk means that critical perception tasks cannot receive sufficient computing power support in a timely manner, leading to delayed responses. To address these issues, this invention proposes a dynamic scheduling scheme for vehicle-mounted visual perception resources based on a safety urgency score. This scheme dynamically reallocates computing resources based on real-time perception of the risk level of the driving scenario, resolving the technical deficiencies of low utilization rates due to fixed computing power and delayed responses in critical scenarios.

[0022] To address the aforementioned technical issues, this invention provides a vehicle-mounted visual perception resource scheduling scheme. This scheme first receives multiple video streams and vehicle status information. The basic perception layer performs primary perception processing on each video stream in a low-power mode, outputting structured primary perception events. Subsequently, the primary perception events and vehicle status information are fused to calculate the safety urgency score of the current driving scenario. Then, based on the safety urgency score and scenario type, the priority configuration of analysis tasks is determined. Finally, based on the priority configuration and the real-time load status of current computing resources, advanced analysis tasks are dynamically allocated to the corresponding computing cores of the NPU and / or GPU for execution, obtaining accurate perception results. This scheme, through a dynamic scheduling mechanism driven by the safety urgency score, ensures that limited computing power always prioritizes perception tasks with the highest safety requirements, a significant difference from the fixed resource allocation schemes in existing technologies.

[0023] Figure 1 A schematic diagram illustrates an example environment in which various embodiments of the present invention can be implemented. For example... Figure 1 As shown, the example environment includes an in-vehicle domain controller and multiple cameras, including a driver monitoring camera, a cabin panoramic camera, a left electronic exterior rearview mirror camera, a right electronic exterior rearview mirror camera, and a rear-view camera. Each camera transmits video data streams to the in-vehicle domain controller via an in-vehicle Ethernet or Low-Voltage Differential Signaling (LVDS) interface. The vehicle also transmits vehicle status information, such as vehicle speed, acceleration, steering wheel angle, turn signal status, gear information, brake pedal status, and door open / close status, to the in-vehicle domain controller in real time via a Controller Area Network (CAN) or CAN FD bus at a cycle of at least 10Hz. The in-vehicle domain controller integrates a System-on-a-Chip (SoC), which contains heterogeneous computing units such as a CPU, NPU, and GPU. This SoC receives multiple video data streams and vehicle status information, executes the in-vehicle visual perception resource scheduling method of this invention, and outputs accurate perception results. The in-vehicle domain controller can also be expanded into an edge computing node or cloud server with corresponding computing capabilities, interacting with the vehicle sensor system via an in-vehicle Ethernet or wireless communication module. The data collection and processing in this application strictly comply with relevant laws and regulations. The collection of driver status images, occupant images, and vehicle location data requires authorization and is used solely for vehicle-mounted visual perception resource scheduling, not for any other unauthorized purposes. It should be understood that the above example environment is described for illustrative purposes only and does not imply any limitation on the scope of the invention. The invention can also be applied to other scenarios such as intelligent connected vehicles and unmanned delivery vehicles, and the environment may also include components such as radar sensors and positioning modules (not shown).

[0024] Figure 2A flowchart of an in-vehicle vision perception resource scheduling method according to some embodiments of the present invention is shown. Example methods may be derived, for example, from... Figure 1 The illustrated vehicle domain controller performs this action. It should be understood that the method may also include additional actions not shown, and the scope of the invention is not limited in this respect. The method includes the following steps: S101: Receive multiple video streams and vehicle status information, perform perception processing on the multiple video streams, and output primary perception events. For example, the vehicle domain controller receives video streams collected by multiple cameras mounted on the vehicle, as well as vehicle status information transmitted in real time via the CAN / CAN FD bus. It then utilizes the lightweight computing power of the SoC's CPU to execute basic visual perception algorithms in parallel on each video stream in low-power mode, encapsulating the detection results into structured primary perception events for output.

[0025] In some embodiments, the multi-channel video data refers to video streams collected separately by multiple cameras mounted on the vehicle, including video of the driver's face and upper body collected by the driver monitoring camera, video of all occupants collected by the cabin panoramic camera, video of the side and rear environment collected by the left and right electronic rearview mirror cameras, and video of the environment behind the vehicle collected by the rear-facing camera. For example, each camera transmits the video data stream to the vehicle domain controller via an in-vehicle Ethernet or LVDS interface, with a video frame rate of not less than 25fps.

[0026] In some embodiments, the vehicle status information refers to vehicle motion status data obtained in real time through CAN / CAN FD bus parsing, including at least vehicle speed, longitudinal acceleration, lateral acceleration, steering wheel angle, turn signal status, gear information, brake pedal status, and door open / close status.

[0027] In some embodiments, the vehicle domain controller performs frame-level time alignment of each video stream through a hardware synchronization trigger signal or a software synchronization mechanism based on precise timestamps, and assigns a globally unified timestamp to each video frame. For example, if the inter-frame time difference of a video stream exceeds the tolerance threshold due to transmission delay, the vehicle domain controller discards the lagging frame or uses nearest neighbor frame interpolation to ensure the time base consistency of subsequent processing.

[0028] In some embodiments, the vehicle domain controller preprocesses the synchronized video frames. Exemplarily, the preprocessing includes resolution scaling, color space conversion, lens distortion correction, and preliminary image enhancement based on histogram equalization. The preprocessed video frames are written to a circular buffer for parallel reading by the underlying vision algorithms.

[0029] In some embodiments, the vehicle domain controller parses vehicle status information in real time and aligns the parsed vehicle status information with video frames in time. For example, the vehicle status information is parsed in real time at a period of not less than 10Hz, and the parsed vehicle status information and video frames are aligned based on the same global timestamp and stored in a status cache queue for subsequent security assessment processing.

[0030] In some embodiments, the vehicle domain controller utilizes the lightweight computing power of the SoC's CPU to execute basic visual perception algorithms in parallel on the preprocessed video streams, and runs continuously in a low-power mode. For example, the basic visual perception algorithms perform low-computational-complexity primary detection and classification tasks, without performing deep feature extraction or temporal prediction.

[0031] Specifically, the vehicle domain controller processes the video stream from the driver monitoring camera using the basic algorithms of the Driver Monitoring System (DMS), performs face detection, eye localization, and head pose angle estimation, and outputs the driver's gaze direction area and eye closure degree.

[0032] Specifically, the vehicle domain controller processes the video stream from the in-cabin panoramic camera through the basic algorithm of the Occupant Monitoring System (OMS), performs key point detection of the human skeleton and seat area occupancy recognition, and outputs the occupant's body outline and posture category.

[0033] Specifically, the vehicle domain controller processes the video streams from the left electronic rearview mirror camera, the right electronic rearview mirror camera, and the rear camera through the basic algorithms of the Camera Monitor System (CMS), performs moving target detection and simple optical flow tracking, and outputs the blind spot target presence status, target category, and pixel-level position.

[0034] In some embodiments, the DMS basic algorithm, OMS basic algorithm and CMS basic algorithm described above run in parallel on the CPU as independent threads without blocking each other, so as to meet the real-time requirements of perception processing of each video stream.

[0035] In some embodiments, the vehicle domain controller encapsulates the detection results of each basic vision algorithm into structured primary perception events at fixed intervals and sends them to subsequent processing flows via an internal message bus. For example, each primary perception event includes fields such as a unique event identifier, an event type code, an event generation timestamp, algorithm confidence, and the coordinates or quantization value of the original detection box. For instance, the event type code includes types such as driver gaze deflection events, driver fatigue level events, abnormal occupant posture events, side / rear target approach events, and blind spot target presence events.

[0036] In some embodiments, the vehicle domain controller continuously outputs structured primary perception events through the aforementioned basic visual perception algorithm. Thus, the basic perception layer runs resident in a low-power mode, continuously outputting structured primary perception events, providing a real-time data foundation for subsequent safety assessments, and achieving continuous perception of the driving scenario without consuming significant NPU / GPU computing resources.

[0037] S102. Based on the primary perception events and vehicle status information, calculate the safety urgency score for the current driving scenario. For example, the vehicle domain controller reads the set of primary perception events and the vehicle status data packet within the current sliding time window, extracts risk factors from valid events, and calculates normalized risk sub-scores for four dimensions: vehicle status, driver status, occupant status, and environmental risk. The vehicle domain controller performs weighted fusion of the risk sub-scores for the four dimensions and dynamically adjusts them according to the risk change trend to obtain the safety urgency score S, and identifies the scenario type of the current driving scenario based on the risk factors.

[0038] In some embodiments, the vehicle domain controller first performs event validity filtering on the primary sensing events within the current sliding time window. Specifically, the vehicle domain controller removes primary sensing events with a confidence level below a preset threshold, as well as expired events whose timestamps differ from the current assessment time by more than a preset time window, and retains the primary sensing events that pass the validity filtering. Through the above processing, the primary sensing events participating in the risk assessment have corresponding timeliness and reliability.

[0039] In some embodiments, the vehicle domain controller extracts risk factors from valid primary perception events and vehicle state data. Specifically, the risk factors include vehicle motion state factors, driver state factors, occupant state factors, and environmental risk factors. For example, vehicle motion state factors include current vehicle speed, longitudinal acceleration, lateral acceleration, steering wheel angle change rate, and turn signal status. For example, driver state factors include gaze deviation direction, gaze deviation duration, and fatigue level. For example, occupant state factors include abnormal posture type, occupant seat number, and seat belt status. For example, environmental risk factors include relative distance and relative speed of side and rear targets, and blind spot target presence markers.

[0040] In some embodiments, the vehicle domain controller identifies the scenario type of the current driving scenario based on the extracted combination of risk factors. For example, the scenario types include scenarios of abnormal driver status, scenarios of approaching targets around the vehicle, blind spot risk scenarios, scenarios of abnormal occupant status, and scenarios involving multiple coupled risks.

[0041] In one possible implementation, the vehicle domain controller classifies the current driving scenario based on whether the current risk factors meet the criteria for a preset scenario type. For example, if the driver's state factors meet the criteria for an abnormal driver state scenario, the current driving scenario is identified as an abnormal driver state scenario. Similarly, if risk factors such as the relative distance and relative speed of targets around the vehicle meet the criteria for a target approach scenario, the current driving scenario is identified as a target approach scenario. In some embodiments, when the current risk factors simultaneously meet the criteria for multiple scenario types, the current driving scenario is identified as a multi-risk coupled scenario.

[0042] The scenario type of the current driving scenario will subsequently serve as the basis for determining the priority configuration of the analysis task in step S103.

[0043] In some embodiments, the vehicle domain controller determines vehicle state risk, driver state risk, occupant state risk, and environmental risk based on extracted risk factors, and quantifies and fuses the risk factors within each risk dimension to obtain corresponding risk sub-scores. In one possible implementation, the vehicle domain controller first converts different types of risk factors into risk quantification values ​​within a unified range. These risk quantification values ​​characterize the degree of influence of the corresponding risk factor on the current driving safety state. For example, for continuously changing risk factors such as vehicle speed, acceleration, relative distance, relative velocity, and duration of line-of-sight deviation, normalization is performed based on the value range of the corresponding risk factor and a preset risk threshold to obtain the corresponding risk quantification value. For example, for state-type risk factors such as fatigue level, abnormal posture type, seatbelt status, and blind spot target presence status, quantification is performed based on the risk level corresponding to different states to obtain the corresponding risk quantification value.

[0044] In some embodiments, the risk quantification value ranges from [0,1], with a larger value indicating a higher risk level for the corresponding risk factor. Subsequently, the vehicle domain controller fuses multiple risk quantification values ​​within the same risk dimension according to the risk dimension to which different risk factors belong, to obtain vehicle status risk score, driver status risk score, occupant status risk score, and environmental risk score, respectively.

[0045] (1) Vehicle condition risk score In one possible implementation, the vehicle domain controller calculates a vehicle state risk score based on the current vehicle speed, the combined vehicle acceleration, and the rate of change of the steering wheel angle. Specifically, the vehicle domain controller performs a weighted summation of the risk items corresponding to the current vehicle speed, the combined vehicle acceleration, and the rate of change of the steering wheel angle to obtain the vehicle state risk score. For example, when the vehicle speed exceeds a preset high-speed threshold and the rate of change of the steering wheel angle is large, the vehicle state risk score increases. Exemplarily, the above-mentioned vehicle state risk score is used to characterize the degree of impact of the vehicle's current motion state on driving safety.

[0046] (2) Driver status risk score In one possible implementation, the vehicle domain controller calculates a driver state risk score based on the duration of driver gaze deviation, the vehicle's current speed, and the driver's fatigue level. Specifically, the vehicle domain controller performs a weighted summation of the coupling risk between the duration of driver gaze deviation and vehicle speed, as well as the risk item corresponding to the driver's fatigue level, to obtain the driver state risk score. For example, when a driver experiences a prolonged gaze deviation while driving at high speed, the driver state risk score is higher than the score corresponding to the same gaze deviation duration while driving at low speed.

[0047] (3) Occupant status risk score In one possible implementation, the vehicle domain controller calculates an occupant status risk score based on occupant posture anomalies and safety constraint states. Specifically, the vehicle domain controller performs a weighted summation of the risk items corresponding to occupant posture anomalies and seatbelt status to obtain the occupant status risk score. For example, when an occupant is not wearing a seatbelt and is leaning forward, the occupant status risk score increases, triggering corresponding perception resource scheduling.

[0048] (4) Environmental risk score In one possible implementation, the vehicle domain controller calculates an environmental risk score based on the time-to-collision (TTC) of the dynamic target and the presence status of the target in the blind spot. Specifically, the vehicle domain controller performs a weighted summation of the risk items corresponding to the dynamic target's TTC and the presence status of the target in the blind spot to obtain the environmental risk score. For example, when the TTC of a side-rear target is lower than a preset danger threshold, the environmental risk score increases, triggering the activation of the corresponding advanced analysis task.

[0049] In some embodiments, the vehicle domain controller obtains the vehicle state risk score, driver state risk score, occupant state risk score, and environmental risk score respectively, and then performs a weighted fusion of the risk sub-scores of the above four dimensions to obtain a comprehensive safety risk score R. The weights corresponding to the risk scores of each dimension satisfy normalization constraints, and the weights are used to reflect the degree of influence of different risk dimensions in the safety assessment of the current driving scenario. In some embodiments, the weights are configured according to vehicle model, usage scenario, or regulatory requirements.

[0050] It is understandable that the comprehensive safety risk score R can characterize the risk level of the current driving scenario. Considering that under the same risk level, scenarios with continuously increasing risk and scenarios with gradually decreasing risk have different levels of urgency, a safety urgency score S is determined by further combining the risk change trend and the time urgency of the dangerous event on the basis of the comprehensive safety risk score R.

[0051] In some embodiments, the vehicle domain controller calculates a safety urgency score S based on the comprehensive safety risk score R, the risk change trend parameter G, and the hazardous event time parameter K:

[0052] Wherein, R represents the comprehensive risk score of the current driving scenario, used to characterize the overall risk level of the current driving scenario; G represents the risk change trend parameter, used to characterize the direction and degree of change of the comprehensive risk score R within a preset time window; K represents the time urgency parameter of dangerous events, used to characterize the time urgency of the identified dangerous events in the current driving scenario; λ is the risk change trend influence coefficient, used to control the degree of influence of G on the safety urgency score S; μ is the dangerous event time influence coefficient, used to control the degree of influence of K on the safety urgency score S.

[0053] The risk change trend parameter G can be determined based on the change between the current comprehensive risk score R and the comprehensive risk score of the previous time or within a preset historical time window. For example, G can be determined according to the following formula:

[0054] in, This represents the overall risk score at the current moment. This represents the comprehensive risk score at the previous sampling time or the start time of a preset time window. This indicates the time window in which the risk score changes. `G` represents the maximum value of the comprehensive risk score, and `clip` represents the processing function that limits the calculation result to a preset range. In this way, when the comprehensive risk score `R` continuously increases, `G` takes a positive value, further increasing the safety urgency score `S` relative to the score determined solely by `R`; when the comprehensive risk score `R` remains relatively stable, `G` approaches 0, weakening its corrective effect on `S`; when the comprehensive risk score `R` continuously decreases, `G` takes a negative value, causing `S` to decrease accordingly.

[0055] In some embodiments, the urgency parameter K of a dangerous event is used to characterize the urgency of the time between the identified dangerous event and its expected occurrence in the current driving scenario. The vehicle domain controller first determines whether there is a predictable dangerous event in the current driving scenario that meets preset conditions based on the current comprehensive risk status; when a predictable dangerous event exists, it predicts the occurrence time of the dangerous event based on at least one of vehicle status information, target motion state, driver state, and / or occupant state, and determines the urgency parameter K of the dangerous event based on the occurrence time of the dangerous event; when no predictable dangerous event exists, it does not predict the occurrence time of the dangerous event, and sets K to a preset baseline value, which can be 0.

[0056] The hazardous events mentioned are not limited to collision events, but may also include at least one of the following: a target entering a preset hazardous area, a vehicle and a target entering a potential conflict state, the driver's state reaching a preset hazardous level, and the occupant's posture reaching a preset hazardous state. For example, when there is a potential collision relationship between a target around the vehicle and the vehicle, the estimated time of the collision event can be predicted based on the distance between the target and the vehicle, the relative speed, and the direction of movement; when a driver's continuous line of sight is detected and there is a further deterioration trend, the estimated time to reach the hazardous state can be determined based on the duration of the line of sight deviation and a preset hazardous state threshold; when there is no hazardous event that meets the preset conditions, K is set to 0.

[0057] In one implementation, when a predictable hazardous event exists, the timing of the event can be determined based on the expected occurrence time. Determine K. For example, it can be calculated using the following formula:

[0058] in, Indicates the predicted time of occurrence of the dangerous event. K represents the preset upper limit of the time reference, and clip represents the processing function that restricts the calculation results to the range of 0 to 1. Therefore, when the expected occurrence time of a dangerous event is long, K is small, and the correction effect of the time urgency of the dangerous event on the safety urgency score S is weak; when a dangerous event is about to occur, K increases, and the correction effect on S is enhanced.

[0059] In some embodiments, λ and μ are calibrated based on actual vehicle operating data and configured according to vehicle model, usage scenario, and / or safety risk strategy. Specifically, when the safety strategy focuses more on driving scenarios where risks rapidly deteriorate, λ can be increased to enhance the influence of the risk change trend parameter G on the safety urgency score S; when the safety strategy focuses more on time-sensitive driving scenarios where dangerous events are about to occur, μ can be increased to enhance the influence of the dangerous event time urgency parameter K on S.

[0060] Using the above method, the safety urgency score S simultaneously considers the current risk level, the trend of risk changes, and the imminent urgency of a dangerous event. Specifically, R characterizes the magnitude of the current risk, G characterizes whether the risk is rapidly deteriorating, and K characterizes whether a dangerous event is imminent. When the current overall risk level is high and the risk continues to rise rapidly, R and G work together to further increase S; when the current overall risk level is high and a dangerous event is imminent, R and K work together to further increase S; when the risk level continues to decrease or there are no predictable dangerous events, the corrective effect of G and / or K weakens, causing S to decrease accordingly.

[0061] In some embodiments, the vehicle domain controller imposes range constraints on the dynamically corrected safety urgency score S, ensuring that the safety urgency score S is within a preset score range.

[0062] In some embodiments, the vehicle domain controller repeatedly performs the above-described safety urgency score calculation process at a fixed evaluation cycle, so that the safety urgency score S is updated as the risk changes in the current driving scenario.

[0063] Therefore, the safety urgency score S not only reflects the overall risk level of the current driving scenario, but also dynamically represents the urgency of the current risk by combining the trend of risk changes and the time urgency of dangerous events.

[0064] It should be noted that the calculation method, weights of each dimension, and dynamic adjustment parameters of the above-mentioned risk sub-scores are merely illustrative descriptions. In specific implementations, the risk quantification method, weight configuration, and dynamic adjustment parameters of each dimension can be adjusted according to actual data, and this application does not impose any limitations on this.

[0065] Step S103: Determine the priority configuration of analysis tasks based on the safety urgency score and the scenario type of the current driving scenario. For example, the vehicle domain controller determines the advanced analysis tasks to be executed in the current driving scenario and the execution priority of each advanced analysis task based on the scenario type of the current driving scenario, the safety urgency score, and the correlation between each advanced analysis task and the current risk factors, and generates a task activation list and priority vector as the basis for subsequent dynamic computing power allocation.

[0066] The scenario type is used to identify the advanced analysis tasks that need to be addressed, the risk correlation is used to characterize the degree of correlation between each advanced analysis task and the current risk factor, and the security urgency score S is used to characterize the urgency of the current risk. As the security urgency score S increases, the advanced analysis tasks with a higher correlation to the current risk factor receive a higher execution priority, thereby enabling computing resources to be further tilted towards tasks related to the current security risk.

[0067] Specifically, such as Figure 3 As shown, step S103 includes the following process.

[0068] S1031. Determine the set of advanced analytics tasks corresponding to the current driving scenario based on the scenario type. The vehicle domain controller determines the set of advanced analytics tasks corresponding to the current driving scenario from a pre-built scenario-task mapping policy library based on the scenario type of the current driving scenario.

[0069] The scenario-task mapping strategy library is a set of pre-established and stored mappings between scenario types and advanced analysis tasks in the vehicle domain controller. It is used to determine the advanced analysis tasks that need further analysis based on the current driving scenario.

[0070] The scenario type is used to characterize the current driving risk state determined by vehicle state information and primary perception events, and can be identified by a preset scenario type code; the advanced analysis task is used to perform further feature extraction, state recognition, target prediction and / or risk analysis for specific perception targets or risk factors in the current driving scenario.

[0071] In one implementation, the scenario-task mapping strategy library includes multiple preset scenario types and a set of advanced analysis tasks corresponding to each scenario type. Different scenario types correspond to different sets of advanced analysis tasks, enabling the vehicle domain controller to select advanced analysis tasks that match the current safety analysis requirements based on the current driving scenario.

[0072] For example, for the "rapid approach of targets around the vehicle" scenario, the scenario-task mapping strategy library can correspond to the target detection enhancement task, the target trajectory prediction task, and the collision risk analysis task; for the "abnormal driver state scenario", it can correspond to the driver's line of sight tracking task, the driver state confirmation task, and the driving behavior prediction task; for the "abnormal occupant state scenario", it can correspond to the occupant posture continuous tracking task and the occupant behavior prediction task.

[0073] After determining the scenario type of the current driving scenario based on vehicle status information and primary perception events, the vehicle domain controller queries the scenario-task mapping strategy library for the advanced analysis task set corresponding to the scenario type, and uses the advanced analysis task set as the candidate task set for further analysis of the current driving scenario.

[0074] For multi-risk coupled scenarios with multiple risk factors, the vehicle domain controller can obtain the advanced analysis task set corresponding to each risk factor, and merge multiple advanced analysis task sets to obtain the comprehensive task set corresponding to the current driving scenario.

[0075] S1032: Based on the risk factors in the current driving scenario, determine the risk correlation between each advanced analysis task in the advanced analysis task set and the risk factors. The risk factors are determined based on vehicle state information and primary perception events, and are used to characterize specific factors leading to safety risks in the current driving scenario, including at least one of target approach, rapid lateral movement of the target, driver line-of-sight deviation, driver fatigue, and abnormal occupant posture.

[0076] The risk correlation degree is used to characterize the importance of advanced analysis tasks in further analyzing, confirming, or predicting corresponding risk factors. The higher the risk correlation degree, the more important the analysis results of the corresponding advanced analysis task are for the judgment or risk management of the current risk factors.

[0077] In one implementation, the vehicle domain controller pre-configures the association relationships between risk factors and advanced analysis tasks in a scene-task mapping policy library, and determines the risk correlation degree corresponding to each advanced analysis task based on the association relationships. The risk correlation degree can be represented by a preset value, correlation level, or ranking relationship. For example, the risk correlation degree can be divided into three levels: high correlation, medium correlation, and low correlation. Specifically, for the risk factor of "rapid approach of a target on the right rear side of the vehicle," the target detection enhancement task, target trajectory prediction task, and collision risk analysis task can be identified as high correlation tasks, while other tasks without a direct analytical relationship to this risk factor can be identified as low correlation tasks. As another example, for the risk factor of "driver's gaze deviation," the driver's gaze tracking task and driver status confirmation task can be identified as high correlation tasks, while other in-cabin analysis tasks without a direct relationship to driver status risk can be identified as low correlation tasks.

[0078] In some embodiments, when an advanced analytics task corresponds to multiple risk factors simultaneously, the vehicle domain controller determines the risk correlation between the advanced analytics task and each risk factor, and determines the comprehensive risk correlation of the advanced analytics task for the current driving scenario based on the presence of the current risk factors. For example, when "a target rapidly approaching from the left rear" and "the driver's gaze deviates from the left rear area" are detected simultaneously while the vehicle is traveling at high speed, the vehicle domain controller determines the correlation between the target trajectory prediction task and the "target rapidly approaching" risk factor, and the correlation between the driver's gaze tracking task and the "driver's gaze deviates" risk factor, and determines the risk correlation of each advanced analytics task in the comprehensive task set based on the two risk factors that currently exist simultaneously.

[0079] For scenarios with multiple coupled risks, the vehicle domain controller can integrate the risk correlations of each risk factor to obtain the comprehensive risk correlation of each advanced analysis task for the current driving scenario, thereby forming the task-risk correlation relationship under the current driving scenario.

[0080] S1033: Determine the execution priority of each advanced analysis task based on the security urgency score and the risk correlation. The risk correlation characterizes how important the task is to the current risk, and the security urgency score S characterizes how urgent the current risk is. Specifically, task priority adjustment rules under different risk urgency levels can be pre-established, ensuring a relatively balanced execution relationship among the advanced analysis tasks when the risk urgency is low; as the security urgency score S increases, the execution priority of advanced analysis tasks with a high correlation to the current risk factor is gradually increased, while the relative execution priority of tasks with a low correlation to the current risk is decreased or maintained.

[0081] For example, in a scenario where targets are rapidly approaching a vehicle, when the urgency level (S) is low, the target detection enhancement task, target trajectory prediction task, and collision risk analysis task can be executed according to the baseline scheduling strategy. When S increases to a medium urgency level, the execution priority of these tasks is increased, making them take precedence over tasks with lower correlation to the current risk. When S further increases to a high urgency level, the execution priority of the highly correlated tasks is given even higher priority, while the relative priority of the low-correlation tasks is further reduced. In other words, the higher the risk correlation and the higher the safety urgency score (S), the higher the execution priority of the corresponding advanced analysis tasks; and as S increases, the priority difference between highly correlated and low-correlation tasks further increases.

[0082] In one implementation, task priorities can be divided into multiple levels, such as P1, P2, and P3, where P1 represents the highest execution priority, P2 represents the medium execution priority, and P3 represents the lowest execution priority. The vehicle domain controller determines the priority level corresponding to each task based on the urgency and relevance of the risk.

[0083] For example:

[0084] The above priority relationships are only used to illustrate how task priorities change under different levels of risk urgency. Specific priority levels and their corresponding rules can be adjusted according to vehicle configuration and safety requirements.

[0085] Finally, the vehicle domain controller generates a task activation list and a priority vector based on the set of advanced analytics tasks corresponding to the current driving scenario and the execution priority of each advanced analytics task. The task activation list represents the advanced analytics tasks that need to be activated and executed in the current driving scenario, and the priority vector represents the relative execution priority among the activated advanced analytics tasks.

[0086] For example, when a rapidly approaching target is detected from the side and rear while the vehicle is traveling at high speed, and the driver's line of sight is also detected to be deviating, the vehicle domain controller can determine the current scenario as a multi-risk coupled scenario and include the target detection enhancement task, target trajectory prediction task, collision risk analysis task, and driver status confirmation task in the task activation list.

[0087] Assuming that the above tasks have a high degree of risk correlation with the current risk, a corresponding high-priority task configuration can be formed when the safety urgency score S is high. For example: Task activation list = {object detection enhancement, object trajectory prediction, collision risk analysis, driver status confirmation, other intelligent tasks}; priority vector = {P1, P1, P1, P1, P3}, where P1 represents the highest execution priority and P3 represents the lower execution priority.

[0088] In low-urgency scenarios, such as when the vehicle is stationary and the cabin is stable, the baseline task priority can be maintained, and non-real-time tasks such as data compression and background learning can be performed. In medium-urgency scenarios, such as when the driver is slightly fatigued or driving at night in rainy weather, the priority of tasks related to the current risk can be increased, and enhanced analysis tasks such as image denoising can be activated as needed. In high-urgency scenarios, such as when a rapidly approaching target is detected during an active lane change, or when a child is detected leaning out of a window, tasks highly related to the current risk, such as target detection, target tracking, and driver eye tracking, can be set as high-priority tasks.

[0089] S104. Based on the priority configuration and computing resource status, dynamically adjust the computing resource allocation of the analysis task, and execute the analysis task based on the adjusted computing resources to obtain accurate perception results. Specifically, the vehicle domain controller determines the computing resources corresponding to each high-level analysis task based on the task activation list and priority vector, combined with the computing resource requirements of each high-level analysis task and the real-time status of the current SoC heterogeneous computing resources, and dynamically adjusts the computing resource allocation according to the task priority, so that high-priority high-level analysis tasks can obtain computing resources that match their computing needs first; subsequently, the vehicle domain controller executes the corresponding high-level analysis task based on the allocated computing resources to further analyze the primary perception events and obtain accurate perception results.

[0090] In some embodiments, the computing resources include a neural network processor (NPU) and a graphics processing unit (GPU) configured in the vehicle-mounted system-on-a-chip (SoC). The vehicle domain controller abstracts the NPU and GPU into independent computing resource pools and determines the mapping relationship between the advanced analytics tasks and the NPU and / or GPU resource pools based on the computational characteristics of each advanced analytics task.

[0091] For advanced analysis tasks that primarily involve neural network model inference, priority can be given to the NPU resource pool for execution, such as target detection enhancement tasks, driver fatigue recognition tasks, and driver state analysis tasks. For advanced analysis tasks that involve a significant amount of image processing, trajectory calculation, or complex numerical calculation, priority can be given to the GPU resource pool for execution, such as target trajectory prediction tasks and collision risk analysis tasks.

[0092] Furthermore, the vehicle domain controller acquires the real-time computing resource status of the NPU resource pool and GPU resource pool, and determines the currently available computing resources for each resource pool based on the real-time computing resource status. In some embodiments, the vehicle domain controller collects NPU utilization, GPU utilization, and resource usage of currently executing tasks at preset intervals through the performance monitoring unit built into the SoC, and determines the currently available computing resources for the NPU resource pool and GPU resource pool respectively based on the preset computing resource capacity of the NPU and GPU. The real-time computing resource status is used to characterize the computing capacity of each computing resource pool to execute new tasks or adjust existing tasks.

[0093] After determining the computing resource type and currently available computing resources for each advanced analytics task, the vehicle domain controller sorts the active advanced analytics tasks according to the priority vector and allocates corresponding computing resources to each advanced analytics task in priority order.

[0094] Specifically, for high-priority advanced analysis tasks, the vehicle domain controller prioritizes meeting their computing resource requirements; for lower-priority advanced analysis tasks, computing resources are allocated only if the remaining computing resources meet their execution requirements. When multiple advanced analysis tasks have the same priority, the vehicle domain controller can determine the resource allocation order of each task based on its computing resource requirements and the available capacity of the current computing resource pool. For example, when a target rapidly approaches the vehicle, step S103 identifies the target detection enhancement task, target trajectory prediction task, and collision risk analysis task as high-priority tasks. Based on the computational characteristics of these tasks, the vehicle domain controller allocates the target detection enhancement task to the NPU resource pool, and allocates the target trajectory prediction task and collision risk analysis task to the appropriate NPU resource pool and / or GPU resource pool. It then determines the computing resource quota for each task based on the available computing resources of the NPU and GPU, thereby prioritizing the advanced analysis tasks related to the current risk.

[0095] In one implementation, when a newly activated high-priority analytics task has a high execution priority and the currently available computing resources are insufficient to meet the computing requirements of the high-priority analytics task, the vehicle domain controller adjusts the resources of the currently executing low-priority analytics task and allocates the released computing resources to the high-priority high-priority analytics task. Adjusting the resources for the low-priority analytics task includes at least one of reducing the task execution frequency, pausing the task, and migrating the task to other suitable computing resources.

[0096] Specifically, reducing the execution frequency of a high-level analysis task refers to decreasing its execution frequency without completely stopping it, thereby reducing the computing resources it consumes per unit of time. For example, for low-priority occupant attitude analysis tasks, their execution frequency can be reduced from a higher frequency to a lower frequency, thus freeing up some NPU or GPU computing resources for high-priority security analysis tasks.

[0097] Pausing a task means stopping the continuous execution of a low-priority analysis task and releasing the computing resources it currently occupies, so that the freed-up computing resources can be used to execute high-priority advanced analysis tasks. Once the current driving risk has decreased, the vehicle domain controller can reactivate the paused advanced analysis task.

[0098] Migrating tasks to other suitable computing resources means that when the current computing resource pool has insufficient available capacity but other computing resource pools have available computing capabilities, high-level analysis tasks that meet the migration conditions are migrated to other computing resource pools for execution, thereby freeing up the computing capabilities of the original computing resource pool.

[0099] For example, when the NPU resource pool is currently under high load, while the GPU resource pool has available computing resources, the vehicle domain controller can migrate low-priority analysis tasks that meet the GPU execution conditions to the GPU resource pool for execution, thereby freeing up NPU resources for high-priority security analysis tasks.

[0100] For uninterrupted computing operations, the vehicle domain controller can perform operations such as reducing execution frequency, pausing, or migrating tasks after the current computing operation is completed, in order to avoid abnormal task execution status.

[0101] In this way, when the computational resource requirements of high-priority security analysis tasks exceed the currently available computational resources, the computational resource usage of low-priority tasks can be adjusted to release corresponding computational resources for high-priority security analysis tasks, thereby achieving dynamic computing power assurance in response to security risks.

[0102] In one implementation, the vehicle domain controller reacquires the current computing resource status according to a preset scheduling cycle, and redetermines the computing resource allocation relationship for each advanced analytics task based on the task activation list, task priority, and computing resource status. The vehicle domain controller triggers rescheduling when at least one of the following occurs: a new advanced analytics task is activated; the priority of the current advanced analytics task changes; the available computing resources in the NPU resource pool or GPU resource pool change; the waiting time of a high-priority advanced analytics task reaches a preset condition; or the current driving scenario changes.

[0103] For example, when a vehicle moves from a normal driving scenario to a scenario where a target is rapidly approaching the vehicle, step S103 regenerates the task activation list and priority vector. The vehicle domain controller then redetermines the allocation of computing resources based on the updated task activation list and priority vector. Once the target risk is eliminated, the priority of the original high-priority task is reduced or the task exits the activation state. The vehicle domain controller then reallocates the released computing resources to allow other intelligent tasks to resume execution.

[0104] In some embodiments, the vehicle domain controller can form a task ready queue of active high-level analytics tasks according to their priority, and submit the high-level analytics tasks in the ready queue to the corresponding computing resource pools for execution based on the real-time available computing resources of the NPU resource pool and GPU resource pool. When a high-priority task enters the ready queue and current resources are insufficient, the vehicle domain controller prioritizes resource adjustments for low-priority tasks so that high-priority tasks can obtain computing resources as quickly as possible.

[0105] After completing the allocation of computing resources, the vehicle domain controller submits each advanced analysis task to the corresponding NPU resource pool and / or GPU resource pool for execution, and performs in-depth analysis of the primary perception events based on the allocated computing resources.

[0106] For example, in scenarios where targets are rapidly approaching around a vehicle, the vehicle domain controller can use the NPU to perform target detection enhancement tasks, further extracting and identifying target features in the initial perception event; use the GPU to perform target trajectory prediction tasks, predicting the future trajectory of the target based on its historical motion state; and perform collision risk analysis tasks based on the target detection results and trajectory prediction results, thereby obtaining accurate perception results such as the target's precise location, trajectory, and collision risk.

[0107] For example, in scenarios where the driver is in an abnormal state, the vehicle domain controller can prioritize allocating NPU computing resources to the driver's gaze tracking task and the driver's state in-depth analysis task, and obtain accurate perception results such as the driver's gaze direction, fatigue state, or abnormal driving behavior based on the advanced analysis results.

[0108] The accurate perception results can be used as input for subsequent driving risk assessment, driving assistance control, or safety control.

[0109] In this way, the vehicle domain controller dynamically allocates heterogeneous computing resources of NPU and GPU according to task priority, task computing resource requirements and SoC real-time computing resource status. When computing resources are insufficient, it releases computing resources by reducing the execution frequency of low-priority tasks, pausing low-priority tasks or migrating tasks, so that high-priority advanced analysis tasks that are highly related to the current security risks can obtain computing resources first, thereby improving the real-time execution capability of critical security analysis tasks under limited computing resources.

[0110] In some embodiments, the method further includes: the vehicle domain controller associating and fusing precise perception results from different advanced analytics tasks, determining the current comprehensive risk status based on the fused perception results, determining the corresponding collaborative control strategy based on the comprehensive risk status, and generating corresponding collaborative control instructions to control the vehicle's alarm devices, display devices, and / or body actuators to perform corresponding safety responses.

[0111] The precise perception results may include information such as driver status, passenger status, the status of targets around the vehicle, and vehicle motion status. For example, the precise perception results may include "the driver's gaze is not directed towards the left rear view area," "a target vehicle is present to the left rear," "the target vehicle is 70 meters away from the vehicle," and "the target vehicle is approaching at a speed of 30 km / h."

[0112] In some embodiments, the vehicle domain controller performs temporal, spatial, and risk correlation on the precise perception results from different advanced analytics tasks to determine whether different precise perception results correspond to the same risk event.

[0113] Among these, temporal correlation is used to determine whether different precise perception results occur simultaneously within a preset time range; spatial correlation is used to determine whether the targets or risk areas corresponding to different precise perception results have a spatial correspondence; and risk correlation is used to determine whether different precise perception results jointly represent the same safety risk. For example, when precise perception results simultaneously include "the driver's gaze is not directed towards the left rear view area" and "there is a rapidly approaching target to the left rear," the multimodal fusion decision-maker can associate these precise perception results as the same potential driving risk event based on their temporal and spatial correspondence. As another example, when "the occupant is located in the left side of the vehicle" and "there is a rapidly approaching target to the left side of the vehicle" are detected, the occupant's position and the state of targets around the vehicle can be correlated to determine whether there is a potential collision risk between the occupant and an external target to the left side of the vehicle. For multiple precise perception results from the same high-level analysis task, they can also be correlated based on their temporal continuity and state change relationships to form a continuous risk event state.

[0114] By using the above method, the precise perception results obtained from different advanced analysis tasks are converted into comprehensive perception information corresponding to specific risk events, thus avoiding independent control of each precise perception result.

[0115] After completing the correlation and fusion of accurate perception results, the vehicle domain controller determines the comprehensive risk status of the current risk event based on the correlated comprehensive perception information.

[0116] In some embodiments, the overall risk status can be determined based on at least one of the following: risk type, risk severity, target location, target motion state, driver status, occupant status, and risk duration. For example, for vehicle-surrounding target risks, the overall risk status can be determined based on the distance between the target and the vehicle, relative speed, direction of motion, and the driver's attention to the target area; for driver status risks, the overall risk status can be determined based on the driver's line of sight, fatigue level, and the vehicle's current motion state; for occupant safety risks, the overall risk status can be determined based on occupant posture, occupant location, and the status of targets surrounding the vehicle.

[0117] In some embodiments, the vehicle domain controller comprehensively judges multiple risk factors according to preset fusion rules and classifies the current risk state into at least one of low-risk, medium-risk, and high-risk states. For example, when only a distant and stable target is detected to be behind the vehicle's left rear, it can be determined as a low-risk state; when a target to the left rear is detected to be continuously approaching and the driver is not observing the corresponding area, it can be determined as a medium-risk or high-risk state; when a target is detected to be rapidly approaching, the driver's line of sight is deviating, and the vehicle is changing lanes, it can be determined as a high-risk state. The fusion rules can be configured according to vehicle safety requirements and can be adjusted according to the actual vehicle model, vehicle functions, and safety control requirements.

[0118] After determining the overall risk status, the vehicle domain controller determines the collaborative control strategy under the current risk status based on the overall risk status and the corresponding risk event types. The collaborative control strategy is used to determine the control methods to be executed under the current risk status and the linkage relationships between different control methods. The control methods include at least one of alarm control, display control, and vehicle body function control.

[0119] Among them, alarm control is used to provide risk warnings to the driver or passengers through voice, visual prompts and / or vibration; display control is used to adjust the display content, display area and / or risk information labels of the vehicle display device; and body function control is used to control the windows, air conditioning and other vehicle actuators to perform preset safety actions.

[0120] The vehicle domain controller selects one or more control methods and combines them based on the overall risk status, enabling different control methods to execute collaboratively for the same risk event. For example, when it is determined that there is a rapidly approaching target to the left rear and the driver has not observed the left rear area, a left rear risk display command, a directional voice prompt command, and a steering wheel vibration command can be generated simultaneously, so that the display, voice, and tactile alarms respond collaboratively to the same risk direction.

[0121] In one implementation, when the overall risk status reaches a preset alarm condition, the multimodal fusion decision-maker generates differentiated alarm control instructions.

[0122] The differentiated alarm control commands can determine the alarm method, alarm content, alarm intensity, and alarm direction based on the comprehensive risk status and the location of the risk target. For example, for a low-risk status, an icon can be displayed or a low-intensity alert can be given; for a medium-risk status, both voice and visual alerts can be output; for a high-risk status, voice, visual, and vibration alerts can be output simultaneously, with increased alarm intensity. The content of the voice alert can be determined based on the risk type, the visual alert can highlight the risk information by adjusting the display status or flashing frequency of the risk icon, and the vibration alert can output vibrations of varying intensities through actuators such as the steering wheel and seat. For example, when a rapidly approaching target is detected to the left rear and the driver has not observed the left rear area, a voice alert corresponding to the left rear risk location can be generated, and the steering wheel or driver's seat can be controlled to output vibration alerts in the corresponding direction.

[0123] In some embodiments, the vehicle domain controller generates display linkage control commands based on the overall risk status and the location of the risk target to adjust the display method of risk information in the vehicle display device. For example, when it is determined that there is a high-risk target to the left rear of the vehicle, the multimodal fusion decision unit can control the streaming media rearview mirror to adjust the display interface, making the rear panoramic view of the vehicle the main display area, and setting a risk target display window in the corresponding risk area to highlight the risk target to the left rear.

[0124] Furthermore, risk target markers, dynamic tracking frames, and / or distance markings can be overlaid on the display screen of electronic exterior rearview mirrors or other vehicle display devices, enabling drivers to quickly determine the location of risk targets and the distance relationship between the targets and the vehicle.

[0125] When the location or motion state of a risk target changes, the multimodal fusion decision-maker can dynamically adjust the position of the risk target identifier, tracking box, and / or distance markers based on the updated accurate perception results.

[0126] In some embodiments, when the overall risk status meets preset body function control conditions, the vehicle domain controller generates a body function linkage control command and sends the control command to the corresponding body control module through the vehicle communication interface. For example, when driver fatigue is detected and preset control conditions are met, a control command can be sent to the air conditioning control module to adjust the air conditioning airflow direction or mode to assist in waking the driver. For control operations involving vehicle actuators such as windows, seats, and air conditioning, corresponding execution conditions and action restrictions can be preset to avoid additional safety risks arising from the control actions themselves.

[0127] After the collaborative control command is generated, the vehicle domain controller or the corresponding vehicle control module controls the alarm device, display device and / or body actuator to perform corresponding actions according to the collaborative control command.

[0128] Furthermore, the vehicle domain controller can obtain the feedback status after the collaborative control is executed, and combine it with the updated accurate perception results to determine whether the current risk status has changed.

[0129] When a persistent risk is detected, the current collaborative control strategy can be maintained or the corresponding control intensity can be increased. When a risk is detected to be reduced, the alarm intensity can be reduced or some display and vehicle functions can be restored. When a risk is detected to be eliminated, the corresponding alarms can be stopped, the highlighted display of the risky target can be canceled, and the normal control state of the corresponding vehicle functions can be restored. For example, when the target on the left rear completes overtaking and leaves the risk area, the multimodal fusion decision-maker determines that the current risk has been eliminated based on the updated accurate perception results. It can then cancel the highlighted display of the risky target on the left rear, stop the corresponding voice and vibration alarms, and restore the normal display state of the relevant display devices.

[0130] In this way, the vehicle domain controller correlates and fuses the precise perception results obtained from different advanced analysis tasks, and selects the appropriate alarm, display and body function control methods according to the integrated risk status after fusion. This enables multiple vehicle execution channels to respond collaboratively to the same risk event, thereby improving the response accuracy and control coordination of vehicle safety assistance functions.

[0131] Figure 4 A schematic block diagram of an in-vehicle visual perception resource scheduling device according to some embodiments of the present invention is shown. The device includes: a primary perception module configured to receive multiple video data streams and vehicle status information, perform perception processing on the multiple video data streams, and output a primary perception event; a safety assessment module configured to calculate a safety urgency score for the current driving scenario based on the primary perception event and the vehicle status information; a priority configuration module configured to determine the priority configuration of an analysis task based on the safety urgency score and the scenario type of the current driving scenario; and a resource scheduling module configured to dynamically adjust the allocation of computing resources for the analysis task based on the priority configuration and computing resource status, and execute the analysis task based on the adjusted computing resources to obtain accurate perception results.

[0132] In some implementations, the resource scheduling module includes a neural network processor (NPU) and a graphics processing unit (GPU) located in the vehicle-mounted system-on-a-chip (SoC), and allocates analysis tasks to the corresponding computing cores of the NPU and / or GPU for execution based on the priority of the analysis tasks. This heterogeneous collaborative scheduling of the NPU and GPU fully utilizes the hardware resources of the SoC, improving overall computing power utilization efficiency.

[0133] In some implementations, the resource scheduling module is also used to adjust the computing resources occupied by the low-priority analysis task and allocate the released computing resources to the high-priority analysis task when the priority of the high-priority analysis task is higher than that of the currently executing low-priority analysis task. This preemptive scheduling capability ensures the system's rapid response capability in the event of sudden changes in risk scenarios.

[0134] This invention also provides a vehicle, which includes a processor and a memory. The memory stores computer instructions, and when the processor executes the computer instructions, the vehicle performs the vehicle-mounted visual perception resource scheduling method of any of the above embodiments. The vehicle can be a passenger car, a commercial vehicle, or an intelligent connected vehicle, and its processor can be an on-board domain controller, and its memory can be a solid-state drive or a hard disk drive.

[0135] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these instructions are executed by a processor, they implement the vehicle-mounted visual perception resource scheduling method described in any of the above embodiments. The storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), or the like.

[0136] This invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-executable instructions. When executed, these computer-executable instructions implement the vehicle-mounted visual perception resource scheduling method of any of the above embodiments. This computer program product can be deployed in an electronic device as a software installation package via network download or local installation.

[0137] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0138] One or more embodiments of the present invention also provide an electronic device that can be used to implement the vehicle-mounted visual perception resource scheduling method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0139] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM) or other volatile memories that do not persist during the power-off period. The computer program may be stored in the ROM. When the processor executes the computer program, it implements the above-described vehicle-mounted visual perception resource scheduling method.

[0140] In some embodiments, the program may be tangibly contained in a computer-readable medium, which may include a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable medium into RAM for execution. The computer-readable medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, or hard disk, storing a computer program that, when executed by a processor, implements the aforementioned vehicle-mounted visual perception resource scheduling method.

[0141] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a server or terminal, they generate, in whole or in part, the processes or functions described in the embodiments of this application. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to the server or terminal, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, and magnetic tape), an optical medium (e.g., digital video disk (DVD), etc.), or a semiconductor medium (e.g., solid-state drive).

[0142] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0143] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for scheduling vehicle-mounted visual perception resources, characterized in that, include: Receive multiple video data streams and vehicle status information, perform perception processing on the multiple video data streams, and output a primary perception event; Based on the primary perceived events and the vehicle status information, a safety urgency score for the current driving scenario is calculated; the safety urgency score is dynamically determined based on the risk level and risk change trend of the current driving scenario. Based on the safety urgency score and the scenario type of the current driving scenario, the priority configuration of the analysis task is determined; Based on the priority configuration and computing resource status, the computing resource allocation of the analysis task is dynamically adjusted, and the analysis task is executed based on the adjusted computing resources to obtain accurate perception results.

2. The vehicle-mounted visual perception resource scheduling method according to claim 1, characterized in that, Based on the safety urgency score and the scenario type of the current driving scenario, the priority configuration of the analysis task is determined, including: Determine the set of advanced analysis tasks corresponding to the current driving scenario based on the scenario type; Based on the risk factors in the current driving scenario, determine the risk correlation between each advanced analysis task in the advanced analysis task set and the risk factors; The execution priority of each advanced analysis task is determined based on the security urgency score and the risk correlation.

3. The vehicle-mounted visual perception resource scheduling method according to claim 2, characterized in that, The execution priority of each of the advanced analysis tasks is determined based on the security urgency score and the risk correlation, including: The relative priority among the advanced analysis tasks is determined based on the risk correlation, and the relative priority is adjusted based on the security urgency score. Among them, advanced analysis tasks with a high risk correlation with the current risk factors are given higher execution priority when the security urgency score increases, while advanced analysis tasks with a low risk correlation with the current risk factors are given lower execution priority.

4. The vehicle-mounted visual perception resource scheduling method according to claim 2, characterized in that, The risk factors are determined based on the vehicle status information and the primary perception events, including at least one of target approach, abnormal target motion state, driver line of sight deviation, driver fatigue, and abnormal occupant posture.

5. The vehicle-mounted visual perception resource scheduling method according to claim 2, characterized in that, The correspondence between the scenario type and the set of advanced analysis tasks, as well as the risk correlation between the risk factors and the advanced analysis tasks, are all pre-stored in the scenario-task mapping strategy library.

6. The vehicle-mounted visual perception resource scheduling method according to claim 1, characterized in that, The computing resources include a neural network processor (NPU) and a graphics processing unit (GPU) configured in the vehicle-mounted system-on-a-chip (SoC). The dynamic adjustment of the computing resource allocation for the analysis task includes: allocating the analysis task to the NPU and / or GPU for execution according to the priority of the analysis task.

7. The vehicle-mounted visual perception resource scheduling method according to claim 6, characterized in that, When a newly activated advanced analysis task has a high execution priority, and the currently available computing resources are insufficient to meet the computing needs of the advanced analysis task, resource adjustments are made to the currently executing low-priority analysis task.

8. The vehicle-mounted visual perception resource scheduling method according to claim 7, characterized in that, Adjusting resources for the low-priority analysis task includes at least one of reducing the execution frequency of the low-priority analysis task, pausing the low-priority analysis task, and migrating the low-priority analysis task to other computing resources.

9. The vehicle-mounted visual perception resource scheduling method according to claim 1, characterized in that, The precise perception results are obtained by the analysis task through further analysis of the primary perception events, and are used to accurately identify target objects, risk states, or driving environments in the current driving scenario.

10. The vehicle-mounted visual perception resource scheduling method according to claim 9, characterized in that, The method further includes: performing fusion analysis on multiple precise perception results, determining the risk level of the current driving scenario based on the fusion analysis results, and generating corresponding collaborative control commands based on the risk level to control the vehicle's display devices, alarm devices, and / or body actuators to perform corresponding linkage operations.

11. A vehicle-mounted visual perception resource scheduling device, characterized in that, include: The primary perception module is configured to receive multiple video data streams and vehicle status information, perform perception processing on the multiple video data streams, and output primary perception events. The safety assessment module is configured to calculate a safety urgency score for the current driving scenario based on the primary perception events and the vehicle status information. The priority configuration module is configured to determine the priority configuration of the analysis task based on the safety urgency score and the scenario type of the current driving scenario; The resource scheduling module is configured to dynamically adjust the allocation of computing resources for the analysis task based on the priority configuration and computing resource status, and execute the analysis task based on the adjusted computing resources to obtain accurate perception results.

12. A vehicle, characterized in that, The system includes a processor and a memory, wherein the memory stores a computer program that, when executed on the processor, causes the vehicle to perform the vehicle-mounted visual perception resource scheduling method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on the processor, it causes the processor to execute the vehicle-mounted visual perception resource scheduling method according to any one of claims 1 to 10.

Images