Vehicle control method, electronic device, and computer-readable storage medium

Abstract

This invention discloses a vehicle control method, an electronic device, and a computer-readable storage medium. The method includes: in response to receiving a vehicle status signal, acquiring original image frames captured by the vehicle and the vehicle's model resource load; determining at least one first inference model corresponding to the vehicle status signal from multiple inference models, and determining the model computing power requirement of the at least one first inference model; allocating resources to the at least one first inference model based on the model resource load and model computing power requirement to obtain first computing resources for the at least one first inference model; inputting the original image frames into the at least one first inference model, and using the at least one first inference model to perform inference based on the first computing resources to obtain an inference result, thereby controlling the vehicle based on the inference result. This invention solves the technical problem of poor vehicle control performance in related technologies.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and more specifically, to a vehicle control method, electronic device, and computer-readable storage medium. Background Technology

[0002] In the field of autonomous vehicle technology, vehicles acquire environmental images through multiple cameras, and the on-board computing system runs various artificial intelligence inference models to achieve perception and decision-making. To support real-time control functions such as surround-view perception, object detection, and behavior prediction, the on-board computing system needs to simultaneously schedule multiple AI inference models to process image frames in parallel. The resource allocation strategies commonly used in related technologies can lead to inference latency and delayed response to critical tasks under high loads, severely affecting the real-time performance and reliability of the on-board computing system, resulting in poor vehicle control performance.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a vehicle control method, an electronic device, and a computer-readable storage medium to at least solve the technical problem of poor vehicle control performance in related technologies.

[0005] According to one aspect of the present invention, a vehicle control method is provided, comprising: in response to receiving a vehicle state signal of a vehicle, acquiring an original image frame captured by the vehicle and a model resource load of the vehicle; determining at least one first inference model corresponding to the vehicle state signal from a plurality of inference models, and determining the model computing power requirement of the at least one first inference model; allocating resources to the at least one first inference model based on the model resource load and the model computing power requirement to obtain a first computing resource for the at least one first inference model; inputting the original image frame into the at least one first inference model, and using the at least one first inference model to perform inference based on the first computing resource to obtain an inference result, so as to control the vehicle based on the inference result.

[0006] Furthermore, when at least one first inference model is multiple first inference models, the original image frame is input into at least one first inference model, and inference is performed using at least one first inference model based on the first computing resources to obtain inference results, including: determining multiple model tasks and task information of multiple model tasks based on the original image frame and vehicle state signals; sorting multiple model tasks based on the task information to obtain a task sorting queue; inputting the original image frame into multiple first inference models, and performing inference on multiple model tasks respectively using multiple first inference models based on the first computing resources and the task sorting queue to obtain inference results.

[0007] Furthermore, the method also includes: in response to the existence of a target model task in the task sorting queue with a waiting processing time longer than a preset time, updating the current priority of the target model task to the target priority, wherein the target priority is higher than the current priority; and updating the task sorting queue based on the target priority.

[0008] Furthermore, the method also includes: in response to receiving an image frame broadcast instruction, determining image frame metadata based on the image frame broadcast instruction; and reading the original image frame from the annular buffer area of ​​the target camera based on the image frame metadata, wherein the annular buffer area is bound to the target camera for storing the image frames acquired by the target camera.

[0009] Furthermore, the method also includes: acquiring multiple raw image frames captured by the target camera; sequentially writing the multiple raw image frames into the memory buffer pointed to by the write pointer in the circular buffer area according to the acquisition time order, wherein the circular buffer area is composed of multiple memory buffer areas, and the write pointer advances to write a memory buffer after writing one image frame.

[0010] Furthermore, the method also includes: in response to detecting a faulty camera among multiple cameras of the vehicle, acquiring identification information and fault information of the faulty camera; sending the fault information to a first vehicle application corresponding to the identification information, wherein the first vehicle application is used to execute corresponding application functions on the original image frames acquired by the faulty camera.

[0011] Furthermore, the method also includes: determining the second inference model corresponding to the identification information from multiple first inference models; and releasing the first computing resources of the second inference model.

[0012] Furthermore, the method also includes: encapsulating the inference result based on the model type of at least one first inference model to obtain a key-value identifier for the inference result; and sending the inference result to the second vehicle application corresponding to the key-value identifier, wherein the second vehicle application is used to execute the corresponding application function according to the inference result.

[0013] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of the present invention during runtime.

[0014] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0015] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0016] According to another aspect of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0017] According to another aspect of the present invention, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of the present invention.

[0018] In this embodiment of the invention, in response to receiving a vehicle status signal, the original image frames acquired by the vehicle and the model resource load of the vehicle are obtained; at least one first inference model corresponding to the vehicle status signal is determined from multiple inference models, and the model computing power requirement of the at least one first inference model is determined; based on the model resource load and model computing power requirement, resources are allocated to the at least one first inference model to obtain the first computing resources of the at least one first inference model; the original image frames are input into the at least one first inference model, and inference is performed using the at least one first inference model according to the first computing resources to obtain the inference result, so as to control the vehicle according to the inference result. This application embodiment is based on a method of dynamically driving inference model selection and adaptive resource allocation based on vehicle status signals. This application first obtains the original image frames acquired by the vehicle and the model resource load of the vehicle, providing a data foundation for the subsequent selection of inference models. Subsequently, at least one inference model and the model computing power requirement are determined from multiple inference models, realizing the accuracy of model selection and the measurability of computing power requirements. By allocating resources to at least one first inference model, this system intelligently matches the required inference model based on vehicle state signals and quantifies the model's computational demands. Accurate resource allocation is then performed in conjunction with system resource load, ensuring that each inference task executes within limited computational resources. Inference can be performed based on the first computational resources, using at least one first inference model to obtain inference results, which can then be used to control the vehicle, improving the real-time response and operational safety of the autonomous driving system in complex situations. This application achieves the goal of ensuring stable inference for high-priority perception tasks in resource-constrained environments, thereby realizing the technical effect of synergistically improving the timeliness of inference response and the accuracy of control decisions, and thus solving the technical problem of poor vehicle control performance in related technologies. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 This is a flowchart of a vehicle control method according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of a vehicle control process according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the specific implementation principle of a vehicle control process according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an optional vehicle control process according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an optional vehicle control process according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of a vehicle control device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of the present invention, a vehicle control method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1 This is a flowchart of a vehicle control method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0030] Step S102: In response to receiving the vehicle status signal, acquire the original image frames collected by the vehicle and the model resource load of the vehicle.

[0031] The aforementioned vehicles refer to intelligent mobile vehicles equipped with multi-sensor perception systems, on-board computing platforms, and autonomous driving control units.

[0032] The aforementioned vehicle status signals refer to structured data signals generated by multiple Electronic Control Units (ECUs) within the vehicle and broadcast via the vehicle's communication bus, used to characterize the vehicle's current operating state. Vehicle status signals encompass multi-dimensional semantic information such as vehicle motion state, control intention, system mode, environmental conditions, and positioning information. Vehicle motion state can refer to vehicle speed, acceleration, steering angle, etc. Control intention can refer to gear position, throttle / brake opening, etc. System mode can refer to the enabled state of autonomous driving, etc. Environmental conditions can refer to light intensity and weather pattern. Positioning information can refer to the accuracy of the Global Navigation Satellite System (GNSS) positioning and map matching confidence level.

[0033] The aforementioned raw image frames refer to the raw pixel data frames directly output by the vehicle's camera sensor without any image preprocessing algorithms. Image preprocessing algorithms can include noise reduction, white balance, distortion correction, gamma correction, or edge enhancement, etc.

[0034] The aforementioned vehicle model resource load refers to the real-time usage status of various hardware resources occupied by the vehicle's onboard computing system at the current moment for executing artificial intelligence inference tasks. The vehicle's model resource load includes, but is not limited to, key indicators such as the computing utilization, memory usage, data bandwidth utilization, cache hit rate, chip temperature, and power consumption of the graphics processing unit (GPU), neural processing unit (NPU), and digital signal processor (DSP).

[0035] In one optional embodiment, during vehicle operation, in response to receiving the vehicle status signal, the on-board computing system synchronously acquires raw image frames generated synchronously by multiple cameras, and dynamically calculates the actual consumption of processor cores, memory bandwidth and cache resources by each inference model based on the scheduling log of the on-board computing system during operation, thereby forming an accurate model resource load.

[0036] This application embodiment obtains original image frames to provide high-fidelity input for subsequent inference, ensuring the accuracy of environmental recognition. Simultaneously, it obtains model resource load to provide a basis for dynamic allocation of computing power. Together, they provide a reliable and efficient data foundation for subsequent vehicle control.

[0037] Step S104: Determine at least one first inference model corresponding to the vehicle state signal from multiple inference models, and determine the model computing power requirement of at least one first inference model.

[0038] The aforementioned reasoning model refers to a deep learning model that has been trained and deployed on an in-vehicle computing platform to perform perception or decision-making tasks, such as object detection models, semantic segmentation models, and behavior prediction models. Its characteristic is that it only performs forward reasoning and does not involve the training process.

[0039] The aforementioned vehicle status signals refer to the dynamic operating parameters collected and broadcast by the vehicle's various electronic control units or sensor systems. Vehicle status signals include, but are not limited to, vehicle speed, gear position, steering angle, light status, Advanced Driver Assistance Systems (ADAS) operating mode, ambient lighting conditions, etc., used to characterize the current driving scenario and functional requirements.

[0040] The aforementioned first inference model refers to at least one inference model selected from a pre-set model library based on the current vehicle state signal that is relatively well matched with the current driving scenario, and is used to perform specific perception or control tasks.

[0041] The aforementioned model computing power requirement refers to the total amount of computing resources required by the inference model when processing a single frame of image. It is usually measured by quantitative indicators such as the number of floating-point operations, memory bandwidth usage, inference latency, or NPU / GPU core utilization rate, and is used to assess the degree to which the model computing power requirement consumes the computing power resources of the in-vehicle system-on-chip (SoC).

[0042] In one optional embodiment, the onboard computing system matches a preset model-signal mapping table with the semantic features of vehicle state signals. This mapping table establishes an association between model function labels and state signal types, thereby selecting at least one semantically consistent first inference model from multiple inference models. Subsequently, the onboard computing system reads information from the first inference model. This information includes network structure complexity, input resolution, operator types, and inference latency constraints. The onboard computing system comprehensively analyzes the number of computing units, memory bandwidth consumption, and cache size required to run at least one first inference model, thereby determining the computing power requirements of at least one first inference model.

[0043] In another optional embodiment, the onboard computing system encodes vehicle status signals into multi-dimensional feature vectors and performs similarity searches in a model library using a lightweight classifier to match at least one suitable first inference model. After determining the first inference model, the onboard computing system uses the static configuration files and historical inference performance logs of at least one first inference model to reverse-engineer the resource consumption pattern of the first inference model under typical inputs, quantifying the dependence on processor cores, memory access frequency, and parallel pipelines, forming a relatively complete description of the model's computing power requirements, and ensuring that the resource allocation module can accurately schedule resources based on actual load characteristics.

[0044] The embodiments of this application accurately match at least one first inference model based on the vehicle status signal and simultaneously obtain the computing power requirements of the first inference model, thereby achieving accurate acquisition of inference tasks and computing resources, avoiding invalid loading and resource waste, and improving the response efficiency and control accuracy of the vehicle computing system.

[0045] Step S106: Based on the model resource load and model computing power requirements, allocate resources to at least one first inference model to obtain the first computing resources of at least one first inference model.

[0046] The aforementioned first computing resource refers to the dedicated computing resource quota dynamically allocated to the first inference model based on the model's resource load and computing power requirements.

[0047] In one optional embodiment, after obtaining the model resource load and model computing power requirements, the vehicle computing system uses a dynamic priority weighting mechanism to perform multi-dimensional matching of the computing power requirements of each first inference model with the current remaining system resources. Based on the model's inference priority, latency sensitivity, and resource dependency, it allocates computing units, memory bandwidth, and cache space proportionally to form dedicated first computing resources, ensuring that the resource allocation process takes into account both fairness and the real-time nature of critical tasks.

[0048] In another optional embodiment, the vehicle-mounted computing system constructs a resource occupancy thermal model based on the real-time distribution map of model resource load, performs a schedulability assessment of the computing power requirements of at least one first inference model, adopts a reservation allocation strategy, reserves an independent computing pipeline and memory buffer for each first inference model, and dynamically adjusts the allocation granularity in combination with load fluctuation prediction, so that the first computing resources can maximize the resource reuse rate while meeting the low threshold of model operation.

[0049] The aforementioned resource occupancy thermal model is a data structure that visualizes and models the real-time load of multimodal computing resources in an in-vehicle computing system based on spatiotemporal dimensions.

[0050] The aforementioned reserved allocation strategy is a resource management method for deterministic real-time inference. The core of the reserved allocation strategy is to statically or semi-statically reserve a dedicated computation pipeline and memory buffer for each first inference model to be scheduled, ensuring that the model can exclusively occupy the required resources when it is triggered for execution, and avoiding scheduling delays, data conflicts or cache pollution caused by multi-task concurrent competition.

[0051] The aforementioned computational pipeline refers to a dedicated hardware execution chain configured for a specific inference model, from data input, preprocessing, model inference to result output. It typically includes an independent instruction flow controller, tensor operation unit, direct memory access (DMA) transfer channel, and result write-back path.

[0052] The aforementioned memory buffer is a contiguous physical memory region bound to the computational pipeline and dedicated to at least one first inference model. It is used to cache input image frames, intermediate feature maps, and the final inference output. The size and access timing of the memory buffer are determined by the model's input size, batching method, and frame rate requirements.

[0053] The aforementioned allocation granularity refers to the smallest schedulable unit used by the onboard computing system when allocating computing resources.

[0054] This application embodiment obtains first computing resources by dynamically allocating resources to at least one first inference model based on the model resource load and computing power requirements, thereby achieving priority scheduling of critical tasks, avoidance of resource contention, and maximization of computing power utilization, thus improving inference response speed and system stability.

[0055] Step S108: Input the original image frame into at least one first inference model, use at least one first inference model to perform inference based on the first computing resources, obtain the inference result, and control the vehicle based on the inference result.

[0056] The aforementioned reasoning result refers to the structured perception or decision data output by the first reasoning model based on the input original image frame and the allocated first computing resources. This structured perception or decision data can be target category, location, speed, lane line coordinates, or hazard warning signals, providing accurate environmental understanding for vehicle control.

[0057] In one optional embodiment, the onboard computing system inputs the original image frame into at least one first inference model that has been allocated first computing resources. The at least one first inference model completes inference operations within the defined computing resource boundaries and outputs structured perception results. Based on the semantic type of the inference results, the onboard computing system encapsulates the inference results into standardized output identifiers and delivers them to the corresponding functional modules via an internal communication bus. The vehicle control unit triggers longitudinal or lateral control commands based on the inference results to achieve closed-loop control of the actuators, ensuring strict synchronization between control actions and perception outputs.

[0058] The aforementioned longitudinal control command refers to the control signal used to control the vehicle's movement along the driving direction (i.e., the forward and backward direction). Its core objective is to adjust the vehicle's acceleration, deceleration, or speed to achieve driving functions such as following other vehicles, adaptive cruise control, emergency braking, or speed maintenance.

[0059] The aforementioned lateral control commands refer to control signals used to control the vehicle's movement perpendicular to the direction of travel (i.e., left and right). Lateral control commands adjust the vehicle's steering angle, yaw rate, or trajectory deviation to achieve functions such as lane keeping, path tracking, automatic lane changing, or cornering cruise control.

[0060] In another optional embodiment, after completing the inference, the on-board computing system performs consistency verification and confidence filtering on the inference results output by at least one first inference model, selecting high-confidence control bases. Then, combined with the contextual information of the vehicle state signals, it generates multi-dimensional control decision signals. These multi-dimensional control decision signals are parsed by the vehicle control protocol stack and transformed into low-level execution instructions, driving the braking, steering, or powertrain systems to perform response actions. This achieves the mapping of inference results to actual vehicle control behavior, ensuring that the control logic maintains high reliability and low latency even under resource-constrained conditions.

[0061] This application's embodiments achieve low-latency response of the perception and control closed loop by accurately mapping the inference results to vehicle control, ensuring that the vehicle's key functions can still be reliably executed under resource constraints, thereby improving the safety of autonomous driving.

[0062] In this embodiment of the invention, in response to receiving a vehicle status signal, the original image frames acquired by the vehicle and the model resource load of the vehicle are obtained; at least one first inference model corresponding to the vehicle status signal is determined from multiple inference models, and the model computing power requirement of the at least one first inference model is determined; based on the model resource load and model computing power requirement, resources are allocated to the at least one first inference model to obtain the first computing resources of the at least one first inference model; the original image frames are input into the at least one first inference model, and inference is performed using the at least one first inference model according to the first computing resources to obtain the inference result, so as to control the vehicle according to the inference result. This application embodiment is based on a method of dynamically driving inference model selection and adaptive resource allocation based on vehicle status signals. This application first obtains the original image frames acquired by the vehicle and the model resource load of the vehicle, providing a data foundation for the subsequent selection of inference models. Subsequently, at least one inference model and the model computing power requirement are determined from multiple inference models, realizing the accuracy of model selection and the measurability of computing power requirements. By allocating resources to at least one first inference model, this system intelligently matches the required inference model based on vehicle state signals and quantifies the model's computational demands. Accurate resource allocation is then performed in conjunction with system resource load, ensuring that each inference task executes within limited computational resources. Inference can be performed based on the first computational resources, using at least one first inference model to obtain inference results, which can then be used to control the vehicle, improving the real-time response and operational safety of the autonomous driving system in complex situations. This application achieves the goal of ensuring stable inference for high-priority perception tasks in resource-constrained environments, thereby realizing the technical effect of synergistically improving the timeliness of inference response and the accuracy of control decisions, and thus solving the technical problem of poor vehicle control performance in related technologies.

[0063] Optionally, when at least one first inference model is multiple first inference models, the original image frame is input into at least one first inference model, and inference is performed using at least one first inference model based on the first computing resources to obtain inference results, including: determining multiple model tasks and task information of multiple model tasks based on the original image frame and vehicle state signals; sorting multiple model tasks based on the task information to obtain a task sorting queue; inputting the original image frame into multiple first inference models, and performing inference on multiple model tasks respectively using multiple first inference models based on the first computing resources and the task sorting queue to obtain inference results.

[0064] The aforementioned multiple model tasks refer to specific perception processing task units derived from multiple first inference models based on the original image frames and vehicle state signals. Although these multiple model tasks all originate from the same image frame, they accomplish different tasks. Some model tasks are responsible for locating pedestrians and vehicles, some for identifying lane lines, some for determining traffic light status, and others for delineating drivable areas. Each task uses a different artificial intelligence model, and the computational load also varies; some are fast and resource-efficient, while others are slow but very complex.

[0065] The task information mentioned above refers to the key data required for the execution of each model task. Task information includes, but is not limited to, the task type, required computing resources, priority, estimated execution time, and the input sources it depends on. The vehicle recomputation system uses this information to determine which task is more urgent and requires fewer resources, thereby rationally scheduling the execution order to ensure that critical tasks are not delayed and that resource allocation is more efficient.

[0066] The task sorting queue mentioned above refers to the execution sequence formed by dynamically sorting multiple model tasks according to dimensions such as priority, timeliness, and dependency in the task information.

[0067] The aforementioned first computing resource refers to the computing resource quota that is pre-allocated to each first inference model during the resource allocation process and cannot be exceeded.

[0068] In one optional embodiment, if the onboard computing system determines that multiple first inference models need to be run simultaneously, it parses several model tasks to be executed based on the original image frames and vehicle status signals, and extracts task information for each task. Task information includes, but is not limited to, priority, computational complexity, and time constraints. The onboard computing system then dynamically constructs a task sorting queue to ensure that higher priority or time-sensitive tasks are placed at the front. Subsequently, the onboard computing system distributes the original image frames to the corresponding inference models. Each model executes its inference task in parallel or in time-sharing according to the pre-allocated first computing resources, and the inference order follows the scheduling instructions of the task sorting queue.

[0069] The embodiments of this application isolate resources and ensure orderly execution between different tasks, avoiding delay fluctuations or task dropping caused by resource competition, and ensuring that each output is generated stably under time consistency and resource controllability, thereby achieving high efficiency and reliability of multi-task collaborative reasoning.

[0070] Optionally, the method further includes: in response to the existence of a target model task in the task sorting queue with a waiting processing time longer than a preset time, updating the current priority of the target model task to the target priority, wherein the target priority is higher than the current priority; and updating the task sorting queue based on the target priority.

[0071] The aforementioned waiting time refers to the cumulative time from when a task enters the queue until the current moment when it has not yet been scheduled for execution, reflecting the degree to which the task is delayed.

[0072] The aforementioned preset duration refers to the maximum allowable waiting time threshold pre-configured during the design phase of the onboard computing system based on the timeliness requirements of the task, the control closed-loop cycle, and the safety redundancy strategy.

[0073] The aforementioned target model task refers to a single inference task that triggers a priority-boosting mechanism due to prolonged waiting in the queue. For example, if the vehicle computing system is simultaneously running lane detection (trigger period 100ms, maximum allowable delay 80ms) and distance monitoring (trigger period 50ms, maximum allowable delay 60ms), and the distance monitoring task has been waiting in the queue for 75ms due to computing power congestion (exceeding the maximum allowable delay of 60ms minus the average time of 20ms minus the safety margin of 5ms, i.e., 60–20–5=35ms), then the distance monitoring task is identified as the target model task. The vehicle computing system will then prioritize the distance monitoring task, inserting it at the front of the queue for execution.

[0074] The aforementioned current priority refers to the initial execution priority level set for the task, which is determined based on the task type and the onboard computing system strategy.

[0075] The aforementioned target priority refers to a new priority that is dynamically increased to avoid long-term task delays. It is higher than the original level and is used for rescheduling.

[0076] The aforementioned update task sorting queue refers to rearranging the order of tasks in the queue according to the new priority.

[0077] In one optional embodiment, if the target model task is stuck at the end of the queue for a long time due to resource shortage while the vehicle computing system is continuously processing multiple tasks, i.e. the waiting time of the above task exceeds the preset time, the vehicle computing system identifies the target task with excessive processing delay, automatically raises the priority of the above target model task to a higher level, and reorders the queue so that the above target model task can be executed in advance.

[0078] The embodiments of this application avoid excessively long information waiting times due to long periods of unprocessed tasks, thereby ensuring the responsiveness of the vehicle computing system to dynamic environments and improving the fairness and real-time performance of overall inference scheduling.

[0079] Optionally, the method further includes: in response to receiving an image frame broadcast instruction, determining image frame metadata based on the image frame broadcast instruction; and reading the original image frame from the annular buffer area of ​​the target camera based on the image frame metadata, wherein the annular buffer area is bound to the target camera for storing image frames acquired by the target camera.

[0080] The aforementioned image frame broadcast command refers to a global request signal issued by other functional modules within the vehicle, used to trigger the recall of historical images from a specific camera, without relying on real-time acquisition.

[0081] The aforementioned image frame metadata refers to the descriptive information carried along with the broadcast command. Image frame metadata may include, but is not limited to, target camera identification, required image time range, frame format requirements, etc., and is used to accurately locate target data.

[0082] The aforementioned target camera refers to a specific image acquisition device in a vehicle multi-camera system that is explicitly designated based on current instructions or task requirements, and which needs to perform data reading, resource binding, or status monitoring.

[0083] The aforementioned circular buffer area is a circular storage structure composed of multiple fixed-size memory buffers, used for efficient caching and continuous rotation of raw image frames captured by the camera. The working principle of the circular buffer area is that the onboard computing system allocates a set of contiguous memory blocks for each camera. Image frames are written sequentially to the current buffer pointed to by the write pointer in the order of acquisition time. After writing is completed, the write pointer automatically advances to the next buffer, and when it reaches the end, it loops back to the first block, thus achieving copy-free, low-latency continuous writing. Simultaneously, multiple downstream modules (such as inference servers, application layers, or fault diagnosis modules) can concurrently read the latest frame or specific historical frames without blocking through read pointers or metadata positioning mechanisms, without copying data or interfering with the writing process. When the aforementioned circular buffer area structure receives an image frame broadcast command, it can accurately backtrack and read images within a specified time period based on timestamps or frame sequence numbers, realizing event replay and fault diagnosis functions. The aforementioned multiple downstream modules can refer to inference servers, application layers, or fault diagnosis modules, etc.

[0084] In one optional embodiment, if the vehicle computing system needs to retrieve recent images from the camera for fault reproduction or behavior analysis, the relevant module issues an image frame broadcast command along with image frame metadata. Based on the camera identifier and time range in the image frame metadata, the vehicle computing system locates the corresponding bound annular buffer area and reverse-reads the required original image frame sequence, without needing to re-acquire images or rely on external storage.

[0085] This application embodiment utilizes the cyclic overlay characteristic of the buffer to ensure that relatively close consecutive images are always retained under limited resources, thereby achieving efficient and low-latency image back-lookup and improving the flexibility and completeness of debugging and anomaly diagnosis of the vehicle computing system.

[0086] Optionally, the method further includes: acquiring multiple raw image frames captured by the target camera; sequentially writing the multiple raw image frames into a memory buffer pointed to by a write pointer in a circular buffer area according to the acquisition time order, wherein the circular buffer area is composed of multiple memory buffer areas, and the write pointer advances to write a memory buffer after writing one image frame.

[0087] The aforementioned write pointer is a dynamic index used in the circular buffer area to track the position of the current image frame to be written. The write pointer works by the in-vehicle computing system initializing a write pointer to the first memory buffer for each camera's circular buffer. Whenever a frame of raw image data is written to the buffer pointed to by the current pointer by the camera driver, the write pointer atomically advances to the next buffer, that is, the pointer address is increased by the buffer size. If the write pointer reaches the end of the buffer sequence, the write pointer value automatically wraps back to the first address, forming a closed loop, thereby achieving continuous data overwriting and uninterrupted writing.

[0088] The aforementioned memory buffer refers to independent, contiguous memory units within a circular buffer region, used to temporarily store single-frame image data.

[0089] In one optional embodiment, the onboard computing system continuously receives raw image frames output by the target camera and writes them one by one into the memory buffer currently pointed to by the write pointer in the circular buffer area according to the acquisition sequence. After each frame is written, the write pointer automatically advances to the next buffer, and when it reaches the end, it returns to the starting position, overwriting the earlier stored image.

[0090] The embodiments of this application ensure that a relatively continuous image sequence is always retained under limited memory, without the need for additional storage management. This not only guarantees the image backtracking capability but also avoids memory overflow, thereby improving the stability and real-time performance of the vehicle computing system in resource-constrained environments.

[0091] Optionally, the method further includes: in response to detecting that a faulty camera exists among the multiple cameras of the vehicle, acquiring identification information and fault information of the faulty camera; sending the fault information to a first vehicle application corresponding to the identification information, wherein the first vehicle application is used to execute corresponding application functions on the original image frames acquired by the faulty camera.

[0092] The aforementioned faulty camera refers to a specific visual sensor in a vehicle's multi-camera system that cannot properly acquire or transmit images due to hardware malfunctions, signal loss, or data anomalies.

[0093] The aforementioned identification information refers to the unique identifier code of the camera, which is used to accurately locate physical or logical nodes in the vehicle architecture and achieve accurate binding of resources and functions.

[0094] The aforementioned fault information refers to structured data describing abnormal camera conditions. Fault information includes, but is not limited to, fault type, occurrence time, and error code, and is used for subsequent diagnosis and response.

[0095] The first vehicle application mentioned above refers to the in-vehicle software module closely related to the functions of the aforementioned cameras, which relies on image input to perform specific tasks. For example, if the side and rear camera malfunctions, the corresponding application is the BlindSpot Detection (BSD) system or the Lane Change Assist (LCA) system, used to determine whether there is a vehicle on the side. If the in-vehicle driver monitoring camera fails, the corresponding application is the Driver Monitoring System (DMS), used to detect fatigue or distraction.

[0096] In one optional embodiment, if the onboard computing system detects an interruption or abnormal output of data from a certain camera, it automatically extracts the identification information and fault type, and sends the above information to the first vehicle application bound to the function. Upon receiving the notification, the first vehicle application can immediately stop waiting for invalid images, switch to a backup strategy, or enter a degraded operation mode to avoid system response delays or misjudgments due to waiting timeouts.

[0097] The embodiments of this application realize the rapid detection of faults and the separation of functional operation, ensuring that other modules do not fail as a whole due to a single point of failure, thereby improving the robustness and fault tolerance of the whole vehicle system.

[0098] Optionally, the method further includes: determining the second inference model corresponding to the identification information from a plurality of first inference models; and releasing the first computing resources of the second inference model.

[0099] The aforementioned second inference model refers to an artificial intelligence inference model that has a functional correspondence with the faulty camera and has been loaded and allocated computing resources in the vehicle computing system.

[0100] In one optional embodiment, if the vehicle computing system detects an anomaly in the camera and reports its unique identifier, it will locate the inference model corresponding to the aforementioned camera from among multiple currently running inference models based on a preset model-camera binding relationship. This second inference model is the one that has been continuously receiving image input from the aforementioned camera and consuming processor and memory resources for inference calculations. Once the vehicle computing system confirms that the input data is no longer sufficient, it will stop the execution of the second inference model and actively reclaim the first computing resources allocated to it.

[0101] The embodiments of this application realize an automated closed loop of fault perception and resource recovery, avoiding the drag on the performance of the vehicle computing system by invalid calculations, and improving the intelligence and timeliness of resource scheduling.

[0102] Optionally, the method further includes: encapsulating the inference result based on the model type of at least one first inference model to obtain a key-value identifier for the inference result; and sending the inference result to the second vehicle application corresponding to the key-value identifier, wherein the second vehicle application is used to execute the corresponding application function according to the inference result.

[0103] The aforementioned key-value identifiers refer to unique tags dynamically generated by the type or functional attributes of the inference model, used to identify the semantic category and purpose of the inference result.

[0104] The aforementioned second vehicle application refers to a software module in the in-vehicle computing system that relies on specific perception results to perform control or decision-making functions. The second vehicle application is a lightweight, decoupled functional unit oriented towards specific perception or control tasks, responsible for consuming the structured reasoning results generated by the first reasoning model and driving the vehicle's decision-making, interaction, or execution behaviors.

[0105] For example, if the inference result indicates that a pedestrian exists 30 meters ahead, the onboard computing system will encapsulate a key-value identifier based on the model type. This identifier is published to the message bus within the vehicle's computing system, where it is detected and responded to by a second vehicle application, triggering a braking command or a visual / voice alarm. Examples of second vehicle applications include automatic emergency braking systems or pedestrian collision warning head-up display systems.

[0106] The core difference between the first and second vehicle applications lies in their functional positioning and triggering mechanisms. The first vehicle application is a low-level module that directly relies on raw image frames from specific cameras to perform perception or control functions. If the corresponding camera malfunctions, the onboard computing system will proactively send an alarm to the application to trigger degradation or emergency strategies. The second vehicle application, on the other hand, is an upper-level module that obtains perceptual semantic information by subscribing to standardized inference results from a key-value server. The second vehicle application does not concern itself with the data source or model details; it only consumes structured output, achieving functional decoupling and reuse. Upper-level modules can refer to automatic parking, driver warning prompts, etc.

[0107] In one optional embodiment, after at least one first inference model completes semantic analysis of the image, the in-vehicle computing system automatically generates a unique key-value identifier based on the model type, serving as a functional label for the inference result. The in-vehicle computing system, through its built-in distribution mechanism, delivers the data packet to a second vehicle application matching the key-value identifier. The in-vehicle computing system has a simple built-in distribution mechanism: after the inference result is generated, the system assigns a unique key-value identifier to the result based on the type of at least one first inference model, then automatically identifies which vehicle applications have subscribed to this identifier. The system then sends the data packet only to these relevant applications, preventing other applications from receiving it and avoiding interference.

[0108] The embodiments of this application realize independent distribution and response of inference and application, improve the independence, maintainability and scalability of the vehicle computing system module, and ensure that the inference results are delivered to the target functional unit efficiently, accurately and without redundancy.

[0109] Figure 2 This is a schematic diagram of a vehicle control process according to an embodiment of this application, such as... Figure 2 As shown, the camera sharing server 202 is responsible for uniformly receiving and caching raw image frames from various cameras in the vehicle, and distributing the image data to the vehicle application layer 201 and the inference server 203 on demand using key-value identifiers. The key-value pair server 205, acting as the core scheduling hub, receives instructions and metadata from the vehicle system layer 206, parses out the target camera and the required inference task, and sends camera fault information from the camera sharing server 202 to the key-value pair server 205. Simultaneously, it maps the task type to the corresponding key-value identifier and forwards it to the inference server 203. Based on the key-value identifiers, the inference server 203 dynamically schedules the matched AI inference engine 204 to perform inference operations, achieving on-demand allocation of model resources. After inference is completed, the AI ​​inference engine 204 sends the result back to the inference server 203, which encapsulates it into a structured output containing key-value identifiers. The key-value pair server then accurately delivers the inference result to the target vehicle application layer 201 module based on the identifiers.

[0110] This embodiment of an in-vehicle multimodal image recognition intelligent assistance system includes an in-vehicle application layer 201, a camera sharing server 202, an inference server 203, a key-value pair server 205, an artificial intelligence inference engine 204, and multiple artificial intelligence models.

[0111] Each AI model corresponds to an application module controlling the vehicle infotainment application layer 201. The AI ​​model is loaded into the vehicle's neural network processor during runtime. The camera sharing server 202 connects to the camera driver module, receives information collected by the camera, and broadcasts image frame data and camera status to modules in the vehicle infotainment application layer 201 and the inference server 203 that subscribe to the camera images. The key-value server 205 receives the camera status published by the camera sharing server 202 through a unified data registry, and also receives soft-switch signals and vehicle status signals from the vehicle infotainment system layer 206 and the vehicle infotainment application layer 201. It stores the received data in identifiers corresponding to the publish-subscribe model. Each identifier stores data from a preset source. When any identifier receives data, the publish-subscribe model pushes the corresponding data to subscribers who subscribe to that identifier. Subscribers include application modules in the vehicle infotainment application layer and the inference server 203.

[0112] The inference server 203 obtains the soft-switching signals and vehicle status signals from the vehicle system layer 206 and the vehicle application layer 201 through the key-value pair server 205. Based on the soft-switching signals and vehicle status signals, it activates the corresponding artificial intelligence model and includes the inference task corresponding to the activated artificial intelligence model in the global inference task queue. The inference server 203 dynamically sorts and allocates computing power for all inference tasks in the queue, then pushes the sorted inference task queue and corresponding image frame data to the artificial intelligence inference engine 204, and receives the standardized inference results returned by the artificial intelligence inference engine 204 and uploads them to the key-value pair server 205.

[0113] The AI ​​inference engine 204 loads the corresponding AI model according to the sorted inference task queue and unloads the AI ​​model from the original unsorted inference task queue.

[0114] The AI ​​inference engine 204 also receives image frame data pushed by the inference server 203 through a standardized interface, uses the image frame data to perform inference operations on the inference task corresponding to each AI model, obtains the standardized results of the category label, confidence, bounding box coordinates and semantic attributes of each AI model, and returns the standardized inference results to the inference server 203.

[0115] The application modules in the vehicle application layer 201 obtain the corresponding artificial intelligence inference results and camera status data by subscribing to the key-value pair server 205.

[0116] In this embodiment, the publish-subscribe model provides multiple key-value identifiers. Each key-value identifier is used to receive and store data from a preset source. Modules in the application layer subscribe to the corresponding key-value identifiers according to their needs. After receiving new data, each key-value identifier pushes or broadcasts the new data to the application layer modules that have subscribed to it.

[0117] In this embodiment, the camera sharing server 202 simultaneously detects whether the camera is faulty or lost. When the camera is faulty or lost, the corresponding status of the camera is published to the key-value pair server 205 in the form of an asynchronous message.

[0118] The key-value pair server 205 sends the camera malfunction or loss information to the inference server 203. The inference server 203 stops activating the artificial intelligence model and sends an alarm signal to the application layer.

[0119] The camera sharing server 202 simultaneously detects whether the camera is faulty or lost. When the camera is faulty or lost, the corresponding status of the camera is published to the key-value pair server 205 in the form of an asynchronous message.

[0120] The key-value pair server 205 sends the camera malfunction or loss information to the inference server 203. The inference server 203 stops activating the artificial intelligence model and sends an alarm signal to the application layer.

[0121] Inference Server 203 includes a dynamic inference decision engine and a resource improvement allocator.

[0122] The dynamic reasoning decision engine obtains soft switch signals and vehicle status signals from the vehicle system layer 206 and vehicle application layer 201 from the key-value pair server 205, activates the corresponding artificial intelligence model based on the soft switch signal and vehicle status signal, and includes the reasoning task corresponding to the activated artificial intelligence model into the global reasoning task queue.

[0123] Based on the metadata configuration file of the activated artificial intelligence model and the computing power status of the system NPU, the resource improvement allocator dynamically sorts and allocates computing power to all inference tasks in the global inference task queue. Then, it pushes the sorted inference task queue and the corresponding image frame data to the artificial intelligence inference engine 204, and receives the standardized inference results returned by the artificial intelligence inference engine 204 and uploads them to the key-value pair server 205.

[0124] The shared memory buffer pool creates five or more buffers for each physical camera driver. These five or more buffers form a circular queue for round-robin storage of raw camera frame data.

[0125] The metadata configuration file for an artificial intelligence model includes priority type, sorting, average inference time, triggering cycle, maximum allowable latency, and load rate.

[0126] The resource improvement allocator ensures that the sum of the computing load of all activated models is less than 90% of the total available computing power of the NPU.

[0127] The resource improvement allocator maintains a real-time updated inference task queue, which is arranged in descending order of priority. When the waiting time of a task is greater than or equal to (maximum allowable delay - average inference time - safety margin), the task is promoted to the front of the queue.

[0128] The standardized inference results output by the AI ​​inference engine 204 are encapsulated into JSON format (JavaScript Object Notation) by the inference server 203 and then sent to the key-value pair server 205.

[0129] This application adopts a three-layer collaborative architecture from camera sharing server to inference server to key-value pair server. Combined with shared memory mechanism, dynamic artificial intelligence model scheduling strategy and unified data bus design, it realizes the whole process improvement from image acquisition, artificial intelligence inference to application decision-making.

[0130] The core of the aforementioned system comprises four main components: a camera sharing server, an inference server, a key-value pair server, and an artificial intelligence inference engine.

[0131] It also supports various upper-layer application modules through standardized interfaces, such as panoramic imaging, blind spot monitoring, driver attention monitoring, forward collision warning, lane departure warning, facial recognition, and other functions.

[0132] Figure 3 This is a schematic diagram illustrating the specific implementation principle of a vehicle control process according to an embodiment of this application, such as... Figure 3 As shown, the camera sharing server sends image frame data to the application layer, while the key-value pair server sends inference results to the application layer. Additionally, other modules in the vehicle system send vehicle status and soft-switch signals to the key-value pair server, and the camera sharing server sends camera fault information to the key-value pair server. Subsequently, the key-value pair server sends vehicle status and soft-switch signals to the inference server, followed by the inference server sending inference results to the key-value pair server. The inference server then sends image frame data model scheduling instructions to the AI ​​inference engine, and finally, the AI ​​inference engine sends inference results to the inference server. The application layer includes, but is not limited to, panoramic imaging, blind spot monitoring, driver attention monitoring, forward collision warning, lane departure warning, and facial recognition.

[0133] The camera sharing server serves as the system's data source and distribution center. It connects to the underlying camera drivers and creates a circular queue of at least five buffers for each physical camera driver, used for round-robin storage of raw camera frame data. This design ensures that when a new frame is written, a complete old frame is always available for the upper-level reader, avoiding read / write conflicts and achieving lossless frame-by-frame retrieval.

[0134] Memory mapping directly maps the memory address space of the aforementioned buffer pool to the process address spaces of various client applications (such as panoramic imaging and blind spot monitoring modules) and the inference server. Other modules can directly read image frames by accessing the mapped pointers, achieving zero-copy data sharing and completely eliminating data copying overhead.

[0135] Image frame data is broadcast in real time to upper-layer applications and inference servers that subscribe to the camera frame data.

[0136] The camera status broadcast monitors the camera driver status in real time and publishes signals such as camera failure and loss to the key-value server in the form of asynchronous messages, which can be subscribed to and consumed by all modules that care about the status.

[0137] The inference server is the system's AI inference and scheduling brain. The dynamic inference decision engine receives soft-switching signals and vehicle status signals from the vehicle system layer and application layer as input conditions. Based on a predefined strategy, it determines which AI models need to be activated at the current moment. This predefined strategy could be to activate lane departure warning only when the vehicle speed exceeds a preset threshold.

[0138] Different artificial intelligence models correspond to different application modules in the application layer. The artificial intelligence model corresponding to the blind spot detection module needs to be activated when the vehicle speed is within the preset threshold, the vehicle is in forward gear, and the blind spot detection soft switch is turned on.

[0139] The artificial intelligence model corresponding to the driver attention monitoring module needs to be activated when the vehicle speed is within a preset threshold and the driver attention monitoring soft switch is turned on.

[0140] The AI ​​model corresponding to the forward collision warning module needs to be activated when the vehicle speed is within a preset threshold, the vehicle is in drive, and the forward collision warning soft switch is turned on.

[0141] The AI ​​model corresponding to the lane departure warning module needs to be activated when the vehicle speed is within a preset threshold and the lane departure warning switch is turned on.

[0142] The AI ​​model corresponding to the facial identifier module needs to be activated when the vehicle speed is 0km / h, not in P gear, and the facial identifier switch is on.

[0143] The resource optimization allocator is the core of scheduling capabilities. It comprehensively senses the available AI computing resources within the system and the resource requirements of the models to be run, implementing a multi-factor-driven dynamic scheduling strategy. AI computing resources can include the number of neural network processor cores, current computing load, memory usage, etc. The key-value server serves as the system's nerve center and data bus. The unified data registry provides a set of registration interfaces, allowing any module within the system to register data that needs to be shared based on predefined key-value identifiers and data formats.

[0144] The publish-subscribe model allows other modules to subscribe to one or more key-value identifiers based on their functional needs. When new data is published to a key, the key-value server immediately pushes or broadcasts the data to all modules that have subscribed to that key. This model replaces messy peer-to-peer communication, making data flow clearer and more efficient.

[0145] In-memory databases store all data in high-speed memory, ensuring extremely low access latency and meeting real-time requirements.

[0146] An AI inference engine is a standard model execution container. It embeds a lightweight deep learning inference model. It receives instructions from the inference server and dynamically loads or unloads specified model files to the neural network processor.

[0147] The standardized interface provides a unified inference interface, receiving image data input and outputting a standardized inference result structure, which is then transmitted to the inference server and distributed via a key-value pair server. It does not contain scheduling logic, thus separating computation from control. The aforementioned inference result structure may include, but is not limited to, category labels, confidence scores, bounding box coordinates, and semantic attributes.

[0148] The application layer module, acting as a resource consumer of the system, consumes data by subscribing to a key-value server to obtain the necessary AI inference results and vehicle signals. Functional logic and alarms execute proprietary logic based on the acquired data, determine the alarm level, and control user interfaces such as sound, light, and user interface icons to provide alarm prompts. The application layer module could include fatigue monitoring, collision warning, etc.

[0149] Figure 4 This is a schematic diagram of an optional vehicle control process according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps:

[0150] Step S402: In response to receiving the vehicle status signal, acquire the original image frames collected by the vehicle and the model resource load of the vehicle.

[0151] Step S404: Determine at least one first inference model corresponding to the vehicle state signal from multiple inference models, and determine the model computing power requirement of at least one first inference model.

[0152] Step S406: Based on the model resource load and model computing power requirements, allocate resources to at least one first inference model to obtain the first computing resources of at least one first inference model.

[0153] Step S408: Based on the original image frame and vehicle state signal, determine multiple model tasks and task information of multiple model tasks; sort the multiple model tasks based on the task information to obtain a task sorting queue; input the original image frame into multiple first inference models, and use the multiple first inference models to perform inference on the multiple model tasks according to the first computing resources and the task sorting queue to obtain inference results, so as to control the vehicle according to the inference results.

[0154] Figure 5 This is a schematic diagram of an optional vehicle control process according to an embodiment of this application, such as... Figure 5 As shown, the process includes the following steps:

[0155] Step S502: In response to receiving the vehicle status signal, acquire the original image frames collected by the vehicle and the model resource load of the vehicle. Step S504: Determine at least one first inference model corresponding to the vehicle status signal from multiple inference models, and determine the model computing power requirement of at least one first inference model.

[0156] Step S506: Based on the model resource load and model computing power requirements, allocate resources to at least one first inference model to obtain the first computing resources of at least one first inference model.

[0157] Step S508: Input the original image frame into at least one first inference model, use at least one first inference model to perform inference based on the first computing resources, obtain the inference result, and control the vehicle based on the inference result.

[0158] Step S510: Based on the model type of at least one first inference model, encapsulate the inference result to obtain a key-value identifier for the inference result; send the inference result to the second vehicle application corresponding to the key-value identifier.

[0159] According to an embodiment of the present invention, a vehicle control device is provided. It should be noted that the device can be used to execute the above-described vehicle control method. Figure 6 This is a schematic diagram of a vehicle control device according to an embodiment of this application, such as... Figure 6 As shown, there is an acquisition module 602, a determination module 604, a resource allocation module 606, and a reasoning module 608.

[0160] The acquisition module 602 is used to acquire the original image frames collected by the vehicle and the model resource load of the vehicle in response to receiving the vehicle status signal; the determination module 604 is used to determine at least one first inference model corresponding to the vehicle status signal from multiple inference models and determine the model computing power requirement of at least one first inference model; the resource allocation module 606 is used to allocate resources to at least one first inference model based on the model resource load and model computing power requirement to obtain the first computing resources of at least one first inference model; the inference module 608 is used to input the original image frames into at least one first inference model, use at least one first inference model to perform inference based on the first computing resources, obtain the inference result, and control the vehicle based on the inference result.

[0161] The inference module is further configured to, when at least one first inference model is multiple first inference models, input the original image frame into at least one first inference model, and use at least one first inference model to perform inference based on a first computing resource to obtain an inference result, including: determining multiple model tasks and task information of multiple model tasks based on the original image frame and vehicle state signal; sorting multiple model tasks based on task information to obtain a task sorting queue; inputting the original image frame into multiple first inference models, and using multiple first inference models to perform inference on multiple model tasks respectively based on the first computing resource and the task sorting queue to obtain an inference result.

[0162] The inference module is also used in the method, which further includes: in response to the existence of a target model task in the task sorting queue with a waiting time longer than a preset time, updating the current priority of the target model task to the target priority, wherein the target priority is higher than the current priority; and updating the task sorting queue based on the target priority.

[0163] The acquisition module is also used to respond to receiving an image frame broadcast instruction, determine image frame metadata based on the image frame broadcast instruction, and read the original image frame from the circular buffer area of ​​the target camera based on the image frame metadata, wherein the circular buffer area is bound to the target camera and is used to store the image frames acquired by the target camera.

[0164] The acquisition module is also used to acquire multiple raw image frames captured by the target camera; and to write the multiple raw image frames sequentially into the memory buffer pointed to by the write pointer in the circular buffer area according to the acquisition time order. The circular buffer area is composed of multiple memory buffer areas, and the write pointer advances to write to a memory buffer after writing one image frame.

[0165] The device is also used to respond to the detection of a faulty camera among multiple cameras in the vehicle, to obtain the identification information and fault information of the faulty camera; and to send the fault information to the first vehicle application corresponding to the identification information, wherein the first vehicle application executes the corresponding application function on the original image frames captured by the faulty camera.

[0166] The device is also used to determine the second inference model corresponding to the identification information from multiple first inference models; and to release the first computing resources of the second inference model.

[0167] The device is also used to encapsulate the inference result based on the model type of at least one first inference model to obtain a key-value identifier for the inference result; and to send the inference result to the second vehicle application corresponding to the key-value identifier, wherein the second vehicle application is used to execute the corresponding application function according to the inference result.

[0168] Embodiments of this application also provide an electronic device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention during runtime.

[0169] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0170] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0171] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of the present invention.

[0172] Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of the present invention.

[0173] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0178] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A vehicle control method, characterized in that, include: In response to receiving the vehicle status signal, the system acquires the original image frames captured by the vehicle and the model resource load of the vehicle. Determine at least one first inference model corresponding to the vehicle state signal from multiple inference models, and determine the model computing power requirement of the at least one first inference model; Based on the model resource load and the model computing power requirement, resource allocation is performed on the at least one first inference model to obtain the first computing resources of the at least one first inference model; The original image frame is input into at least one first inference model, and the at least one first inference model is used to perform inference based on the first computing resources to obtain an inference result, so as to control the vehicle based on the inference result.

2. The method according to claim 1, characterized in that, When the at least one first inference model is multiple first inference models, the original image frame is input into the at least one first inference model, and inference is performed using the at least one first inference model based on the first computing resources to obtain an inference result, including: Based on the original image frame and the vehicle state signal, multiple model tasks and task information of the multiple model tasks are determined. Based on the task information, the multiple model tasks are sorted to obtain a task sorting queue; The original image frame is input into the plurality of first inference models. The plurality of first inference models are used to perform inference on the plurality of model tasks according to the first computing resources and the task sorting queue, so as to obtain the inference result.

3. The method according to claim 2, characterized in that, The method further includes: In response to the presence of a target model task in the task sorting queue with a waiting time longer than a preset time, the current priority of the target model task is updated to the target priority, wherein the target priority is higher than the current priority; The task sorting queue is updated based on the target priority.

4. The method according to claim 1, characterized in that, The method further includes: In response to receiving an image frame broadcast instruction, determine image frame metadata based on the image frame broadcast instruction; Based on the image frame metadata, the original image frame is read from the annular buffer area of ​​the target camera, wherein the annular buffer area is bound to the target camera and is used to store the image frames acquired by the target camera.

5. The method according to claim 4, characterized in that, The method further includes: The target camera captures multiple raw image frames; Multiple original image frames are sequentially written into the memory buffer pointed to by the write pointer in the circular buffer area according to the acquisition time order. The circular buffer area is composed of multiple memory buffer areas, and the write pointer advances to write to a memory buffer after writing one image frame.

6. The method according to claim 1, characterized in that, The method further includes: In response to the detection of a faulty camera among multiple cameras in the vehicle, the identification information and fault information of the faulty camera are obtained; The fault information is sent to the first vehicle application corresponding to the identification information, wherein the first vehicle application is used to execute the corresponding application function for the original image frames captured by the faulty camera.

7. The method according to claim 6, characterized in that, The method further includes: Determine the second inference model corresponding to the identification information from multiple first inference models; Release the first computing resources of the second inference model.

8. The method according to claim 1, characterized in that, The method further includes: Based on the model type of the at least one first inference model, the inference result is encapsulated to obtain a key-value identifier for the inference result; The inference result is sent to the second vehicle application corresponding to the key-value identifier, wherein the second vehicle application is used to execute the corresponding application function based on the inference result.

9. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method according to any one of claims 1 to 8.

Images