Image processing method and system and vehicle

Abstract

The invention discloses an image processing method and system and a vehicle, and relates to the technical field of image processing, and the image processing method comprises the steps: dividing a target image processing task into a plurality of subtasks; according to the inherent characteristic information of each core processor in the multi-core chip, determining a target core processor corresponding to each sub-task in each core processor, and distributing each sub-task to the respective corresponding target core processor; each core processor performs data interaction based on a shared memory space set by the multi-core chip so as to execute each subtask in parallel; wherein each core processor comprises at least two of a central processor, a graphics processor, a neural network processor and a real-time processor. According to the invention, the image processing efficiency can be improved, so that the real-time performance of image processing is ensured.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, system and vehicle. Background Technology

[0002] With the increasing popularity of in-vehicle assistance systems, in-vehicle chips need to process high frame rate video streams from multiple high-definition cameras, and place extremely high demands on real-time performance, power consumption and reliability. Image processing algorithms have become a key factor affecting system performance.

[0003] Currently, hyperasynchronous architecture chips are typically deployed to implement image processing. However, when processing a complete image processing task, multiple subtasks are scheduled to be executed serially on a single core. This not only fails to fully utilize the heterogeneous computing capabilities of the chip, but also causes the processing pipeline to be lengthy due to serial execution and frequent data copying, resulting in excessive load on a single core and increased memory consumption. This leads to increased data processing latency and affects the real-time performance of image processing. Summary of the Invention

[0004] The main objective of this application is to provide an image processing method, system, and vehicle, which aims to improve image processing efficiency and ensure real-time image processing.

[0005] This application provides an image processing method, the method comprising: Break down the target image processing task into multiple sub-tasks; Based on the inherent characteristics of each core processor in the multi-core chip, the target core processor corresponding to each subtask in each core processor is determined, and each subtask is assigned to its corresponding target core processor. Each of the core processors interacts with data based on the shared memory space set up by the multi-core chip in order to execute each of the subtasks in parallel; Each of the core processors includes at least two of the following: a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor.

[0006] In one embodiment, the step of determining the target core processor corresponding to each subtask in each of the core processors based on the inherent characteristic information of each core processor in the multi-core chip includes: For any of the core processors, the target task type corresponding to the functional specialization of the core processor is determined based on the inherent characteristic information of the core processor. The core processor is designated as the target core processor for each of the subtasks whose task type is the target task type.

[0007] In one embodiment, the target task type is a logic control type, an image rendering type, an image acceleration processing type, or a system state management type; The step of designating the core processor as the target core processor for the subtask whose task type is the target task type in each of the subtasks includes: The central processing unit is used as the target core processor for the subtask whose task type is the logic control type. And / or, the graphics processor is used as the target core processor for the subtask whose task type is the image rendering type in each of the subtasks; And / or, the neural network processor is used as the target core processor for the subtask whose task type is the image acceleration processing type in each of the subtasks; And / or, the real-time processor is used as the target core processor for the subtask whose task type is the system state management type in each of the subtasks.

[0008] In one embodiment, prior to the step of assigning each of the subtasks to its corresponding target core processor, the method further includes: Obtain historical processing latency data and load change trends for each of the core processors; Based on the historical processing latency data and load change trends of each core processor, predict the future computing power requirements of each core processor; The proportion of tasks allocated to each core processor is adjusted based on the future computing power requirements of each core processor.

[0009] In one embodiment, the step of adjusting the proportion of tasks allocated to each of the core processors based on the future computing power demand of each core processor includes: For any of the core processors, the target load state that the core processor is ready to enter is determined based on the future computing power requirements of the core processor. If the target load state is a high load state, then reduce the proportion of tasks allocated to the core processor; If the target load state is low load, then increase the proportion of tasks allocated to the core processor.

[0010] In one embodiment, the method further includes: In the case of multiple image processing tasks to be processed, the processing priority of each image processing task to be processed is obtained; Based on the processing priority of each of the image processing tasks to be processed, the target image processing task is determined from each of the image processing tasks to be processed in descending order of priority, and the step of splitting the target image processing task into multiple sub-tasks is executed. After the target image processing task is completed, the process returns to the step of determining the target image processing task from the image processing tasks to be processed in descending order of priority, until all the image processing tasks to be processed are completed.

[0011] In one embodiment, the method further includes: Monitor whether any abnormalities occur during the processing of the target image processing task; If so, obtain abnormal information during the processing of the target image processing task and report the abnormal information.

[0012] In addition, to achieve the above objectives, this application also provides an image processing system, which includes a multi-core chip. Each core processor in the multi-core chip includes at least two of a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor, and the multi-core chip is provided with a shared memory space. The image processing system is used to implement the steps of the image processing method described above.

[0013] In one embodiment, each of the core processors communicates via an IPC mechanism, and each process in each core processor communicates via COOL, while each thread communicates via DDK.

[0014] In addition, to achieve the above objectives, this application also provides a vehicle, which includes a vehicle body and an image processing system as described above, wherein the image processing system is disposed on the vehicle body.

[0015] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image processing method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image processing method described above.

[0017] This application provides an image processing method, which firstly divides the target image processing task into multiple subtasks; then, based on the inherent characteristics of each core processor in a multi-core chip, determines the target core processor corresponding to each subtask in each core processor, and assigns each subtask to its corresponding target core processor; each core processor will interact with data based on the shared memory space set by the multi-core chip to execute each subtask in parallel; wherein, each core processor includes at least two of the following: central processing unit, graphics processing unit, neural network processor, and real-time processor.

[0018] Therefore, the technical solution provided in this application, after breaking down the target image processing task into multiple sub-tasks, flexibly allocates each sub-task by combining the inherent characteristics of each core processor in the multi-core chip. This ensures that the sub-tasks are distributed across the core processors, avoiding excessive load on a single core and fully utilizing the computing power of the multi-core chip, thereby improving image processing efficiency. Furthermore, each core processor interacts with data through the shared memory space set up by the multi-core chip to execute each sub-task in parallel. This not only reduces unnecessary memory usage but also lowers the latency caused by frequent data copying.

[0019] In summary, the technical solution provided in this application can improve image processing efficiency and ensure the real-time performance of image processing. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram illustrating the process of image processing using a hyperasynchronous architecture chip, as provided in an embodiment of this application. Figure 2 A schematic flowchart of the image processing method provided in the first embodiment of this application; Figure 3 This is a schematic diagram illustrating the implementation principle of the image processing method provided in the first embodiment of this application. Figure 4 This is a schematic diagram of the predictive scheduling mechanism module provided in the second embodiment of this application; Figure 5 This is a schematic diagram illustrating the implementation principle of the image processing method provided in this application when applied to a vehicle. Figure 6 The diagram illustrates the deployment tasks and signal flow interaction of each core processor provided in the embodiments of this application.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] With the increasing popularity of in-vehicle assistance systems, in-vehicle chips need to process high frame rate video streams from multiple high-definition cameras, and place extremely high demands on real-time performance, power consumption and reliability. Image processing algorithms have become a key factor affecting system performance.

[0027] Currently, image processing is typically implemented using hyperasynchronous architecture chips, and the specific processing flow can be found in [reference needed]. Figure 1 Specifically, after the video stream arrives, it will be broken down into four sub-tasks: image encoding / decoding, image preprocessing, image denoising and enhancement, and accelerated convolutional processing (i.e., object detection). In these four sub-tasks, image frames from each sub-process will be sequentially queued into buffers 1, 2, and 3. This not only fails to fully utilize the chip's heterogeneous computing capabilities but also results in a lengthy processing pipeline due to serial execution and frequent data copying. This leads to excessive load on a single core and increased memory consumption, thereby increasing data processing latency, affecting the real-time performance of image processing, and ultimately impacting driving / parking safety.

[0028] Based on this, this application provides an image processing method, which firstly divides the target image processing task into multiple subtasks; then, based on the inherent characteristics of each core processor in the multi-core chip, determines the target core processor corresponding to each subtask in each core processor, and assigns each subtask to its corresponding target core processor; each core processor will interact with data based on the shared memory space set by the multi-core chip to execute each subtask in parallel; wherein, each core processor includes at least two of the following: central processing unit, graphics processing unit, neural network processor, and real-time processor.

[0029] Therefore, the technical solution provided in this application, after breaking down the target image processing task into multiple sub-tasks, flexibly allocates each sub-task by combining the inherent characteristics of each core processor in the multi-core chip. This ensures that the sub-tasks are distributed across the core processors, avoiding excessive load on a single core and fully utilizing the computing power of the multi-core chip, thereby improving image processing efficiency. Furthermore, each core processor interacts with data through the shared memory space set up by the multi-core chip to execute each sub-task in parallel. This not only reduces unnecessary memory usage but also lowers the latency caused by frequent data copying.

[0030] In summary, the technical solution provided in this application can improve image processing efficiency and ensure the real-time performance of image processing.

[0031] The execution subject of the image processing method of this application can be an electronic device with data processing, network communication and program running functions, or a control system, control circuit, etc. that can realize the above functions, or an image processing system, vehicle, etc. This embodiment does not specifically limit it.

[0032] The following description uses an image processing system as the execution subject to illustrate the various embodiments.

[0033] This application presents an image processing method according to a first embodiment. Please refer to [link / reference]. Figure 2 The image processing method may include steps S10 to S30: Step S10: Divide the target image processing task into multiple sub-tasks; It should be noted that the target image processing task is the image processing task that the image processing system needs to process at the current moment. The image processing task usually includes multiple steps such as decoding, preprocessing, denoising, and color correction. Subtasks refer to logically independent and executable computational units obtained after decomposing the target image processing task, such as image decoding, image preprocessing, accelerated image convolution processing (object detection), image rendering and stitching, and system status monitoring.

[0034] Step S20: Based on the inherent characteristics of each core processor in the multi-core chip, determine the target core processor corresponding to each subtask in each core processor, and assign each subtask to its corresponding target core processor. It's important to note that the inherent characteristics of a core processor refer to objective and unchanging technical attributes determined by its hardware architecture. Examples include the complex control logic units of the Central Processing Unit (CPU), the numerous parallel computing cores of the Graphics Processing Unit (GPU), the dedicated tensor computation units of the Neural Processing Unit (NPU), and the fast interrupt response mechanism of the Real-time Processing Unit (RPU). These inherent characteristics determine the types of tasks the core processor excels at handling. In practical applications, multiple core processors can be deployed on the same PCB (Printed Circuit Board) to achieve efficient collaborative computing, thereby optimizing overall performance and balancing power consumption and cost.

[0035] In one feasible implementation, step S20 may include steps S21-S22: Step S21: For any core processor, determine the target task type corresponding to the core processor's functional specialization based on the core processor's inherent characteristic information. It should be noted that the functional specialization of a core processor is a summary of a certain type of computational task that it can be designed to execute optimally, directly determined by the inherent characteristics of the core processor. For example, the functional specialization of a graphics processing unit (GPU) is massively parallel floating-point computation, and the functional specialization of a neural network processor is accelerated neural network inference. The target task type is the type of task that matches the functional specialization of the core processor.

[0036] Step S22: Select the core processor as the target core processor for each subtask whose task type is the target task type.

[0037] It should be noted that the target task type corresponding to the functional specialty of the central processing unit is logic control, so logic control tasks can be assigned to the central processing unit for execution; the target task type corresponding to the functional specialty of the graphics processing unit is image rendering, so image rendering tasks can be assigned to the graphics processing unit for execution; the target task type corresponding to the functional specialty of the neural network processor is image acceleration processing, so image acceleration processing tasks can be assigned to the neural network processor for execution; and the target task type corresponding to the functional specialty of the real-time processor is system state management, so system state management tasks can be assigned to the real-time processor for execution.

[0038] Among them, logic control type tasks may include, but are not limited to, logic control tasks, lightweight computing tasks, etc.; image rendering type tasks may include, but are not limited to, image rendering tasks, image stitching tasks, etc.; image acceleration processing type tasks may include, but are not limited to, convolutional neural network acceleration tasks, image convolution processing tasks, etc.; system state management type tasks may include, but are not limited to, real-time signal sending and receiving tasks, intra-process communication or state management tasks, etc.

[0039] In this embodiment, for any core processor, the target task type corresponding to the core processor's functional specialization is first determined based on the core processor's inherent characteristic information. Then, the core processor is designated as the target core processor for each subtask whose task type matches the target task type. This allows each subtask to be assigned to the most suitable core processor for execution, enabling each core processor to achieve its maximum performance and effectively improving image processing efficiency.

[0040] In step S30, each core processor interacts with data based on the shared memory space set up by the multi-core chip in order to execute each subtask in parallel; Each core processor includes at least two of the following: a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor.

[0041] Understandably, by setting up a shared memory space in a multi-core chip, threads in each core processor can directly interact with the same shared memory space. Compared to conventional methods where each process relies on an independent buffer and requires multiple data copies for communication, this embodiment fundamentally reduces the number of data copies, ensuring efficient and real-time data interaction.

[0042] For example, to aid in understanding the implementation principle of the image processing method provided in this embodiment, we will take the core processors, including a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor, as examples. Please refer to... Figure 3 , specifically: After the target image processing task (i.e., video stream) is broken down into multiple subtasks, logic control type subtasks are assigned to the central processing unit (CPU), image rendering type subtasks are assigned to the graphics processing unit (GPU), image acceleration type subtasks are assigned to the neural network processor (NN), and system state management type subtasks are assigned to the real-time processor (RTP). The CPU, GPU, NN, and RTP interact with each other through shared memory space to execute each subtask in parallel, thereby achieving the processing of the target image processing task.

[0043] Based on the above, this embodiment provides an image processing method. First, the target image processing task is divided into multiple sub-tasks. Then, based on the inherent characteristics of each core processor in the multi-core chip, the target core processor corresponding to each sub-task in each core processor is determined, and each sub-task is assigned to its corresponding target core processor. Each core processor will interact with data based on the shared memory space set by the multi-core chip to execute each sub-task in parallel. Among them, each core processor includes at least two of the following: central processing unit, graphics processing unit, neural network processor, and real-time processor.

[0044] Therefore, the technical solution provided in this embodiment, after breaking down the target image processing task into multiple sub-tasks, flexibly allocates each sub-task by combining the inherent characteristics of each core processor in the multi-core chip. This ensures that each sub-task is distributed across the core processors, avoiding excessive load on a single core and fully utilizing the computing power of the multi-core chip, thereby improving image processing efficiency. Furthermore, each core processor interacts with data through the shared memory space set up by the multi-core chip to execute each sub-task in parallel. This not only reduces unnecessary memory usage but also lowers the latency caused by frequent data copying.

[0045] In summary, the technical solution provided in this embodiment can improve image processing efficiency and ensure the real-time performance of image processing.

[0046] Based on the first embodiment described above, a second embodiment of the image processing method of this application is proposed. In the second embodiment, before step S20, the image processing method may further include steps S01 to S03: Step S01: Obtain historical processing latency data and load change trends for each core processor; It should be noted that historical processing latency data refers to the actual time spent by the core processor in processing one or more subtasks over a past period. Load change trends characterize the movement and pattern of the core processor's load over time, and can be used to predict future load trends. Load change trends can include increases, decreases, or stabilization.

[0047] Step S02: Based on the historical processing latency data and load change trends of each core processor, predict the future computing power demand of each core processor; It should be noted that the future computing power demand of the core processor refers to the total amount of computing resources required by the core processor in the near future to meet its task processing requirements.

[0048] Step S03: Adjust the proportion of tasks allocated to each core processor based on the future computing power demand of each core processor.

[0049] It should be noted that the task ratio refers to the proportion of the total number of subtasks to be processed that are allocated to a specific core processor.

[0050] Additionally, it should be noted that the image processing system may include a predictive scheduling mechanism module to execute the above steps S01-S03. Please refer to [reference needed]. Figure 4 The predictive scheduling mechanism module can further include a historical latency statistics module, a load change trend prediction module, and a task scheduling module. Step S01 can be jointly executed by the historical latency statistics module and the load change trend prediction module, while steps S02 and S03 can be executed by the task scheduling module. In practical use, this predictive scheduling mechanism module can be deployed in the central processing unit (CPU). Based on this, the CPU can uniformly classify and aggregate the tasks being processed by other core processors, and can pre-allocate important tasks and processes to appropriate core processors based on the current system load and bandwidth throughput. For example, taking image frame processing as an example, the raw data of the video stream is transmitted to the chip's MIPI (Mobile Industry Processor Interface) via a serializer and then accesses the predictive scheduling mechanism module. Subsequently, this predictive scheduling mechanism module can combine historical processing latency data and load change trends to rationally allocate and schedule important processes in the multi-core chip's image stream processing, ensuring that the video stream data can operate efficiently and effectively within the relevant processes of the chip. Finally, the perception module completes target recognition, and the image display module outputs the data to the vehicle's infotainment screen for display.

[0051] In one feasible implementation, step S03 may include steps S031 to S033: Step S031: For any core processor, determine the target load state that the core processor is ready to enter based on the future computing power requirements of the core processor. In one feasible implementation, step S031 may include: calculating the ratio of the future computing power demand of the core processor to the theoretical peak computing power of the core processor to obtain the computing power demand utilization rate of the core processor; determining whether the computing power demand utilization rate is greater than the actual computing power utilization rate of the core processor at the current moment; if so, determining that the target load state that the core processor is ready to enter is a high load state; if not, determining that the target load state that the core processor is ready to enter is a low load state.

[0052] The theoretical peak computing power refers to the maximum computing power that a core processor can achieve under ideal conditions (i.e., without considering any limiting factors in actual applications), and it is usually measured in floating-point operations per second.

[0053] In another feasible implementation, step S031 may include: calculating the difference between the theoretical peak computing power of the core processor and the future computing power demand of the core processor to obtain the predicted remaining computing power of the core processor; determining whether the predicted remaining computing power is less than a preset computing power threshold; if so, determining that the target load state that the core processor is ready to enter is a high load state; if not, determining that the target load state that the core processor is ready to enter is a low load state.

[0054] The preset computing power threshold serves as the basis for determining whether the remaining computing power is sufficient. It can be a default value or can be flexibly set by the user according to the actual situation. This embodiment does not impose any specific limitations on it.

[0055] The above are only two feasible implementations of step S031 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S031.

[0056] Step S032: If the target load state is a high load state, reduce the proportion of tasks allocated to the core processor. Step S033: If the target load state is low load, increase the proportion of tasks allocated to the core processor.

[0057] In this embodiment, the target load state for the core processor to enter is determined by utilizing the future computing power demand of the core processor. If the target load state is a high load state, it indicates that the core processor will experience a shortage of computing resources in the future, i.e., an excessively high load. In this case, the proportion of tasks allocated to the core processor can be reduced in advance to effectively prevent the problem of increased processing latency caused by overload. If the target load state is a low load state, it indicates that the core processor will have sufficient computing resources in the future. In this case, the proportion of tasks allocated to the core processor can be increased in advance to effectively avoid the waste of idle computing resources and improve the overall throughput and task completion efficiency of the system.

[0058] In another feasible implementation, step S03 may include: obtaining the task ratio corresponding to the future computing power demand of each core processor based on the preset mapping relationship between computing power demand and task ratio, as the initial task ratio corresponding to each core processor; normalizing the initial task ratio corresponding to each core processor to obtain the target task ratio corresponding to each core processor, and adjusting the task ratio allocated to each core processor to its corresponding target task ratio.

[0059] Among them, the computing power demand is negatively correlated with the task ratio, that is, the greater the computing power demand, the smaller the task ratio; the smaller the computing power demand, the larger the task ratio.

[0060] The above are only two feasible implementation methods of step S03 provided in this embodiment. This embodiment does not specifically limit the specific implementation method of step S03.

[0061] Understandably, current task scheduling mechanisms suffer from timing lag, typically triggering only when the system is already overloaded. Since the system is already overloaded when the task scheduling mechanism is triggered, significant processing latency is inevitably introduced. To address this, this embodiment configures the system to predict the future computing power requirements of each core processor in advance, based on historical processing latency data and load trends, before allocating each subtask to its corresponding target core processor—that is, before the image enters the processing pipeline. This allows for the adjustment of the task allocation ratio to each core processor. Thus, the task allocation ratio can be proactively adjusted before system overload occurs, ensuring the stability of the image processing flow from the outset. This guarantees that the system can maintain stable throughput and low latency under sudden high load scenarios, avoiding performance drops due to resource contention.

[0062] Based on the first and / or second embodiments described above, a third embodiment of the image processing method of this application is proposed. In the third embodiment, the image processing method may further include steps S100 to S300: Step S100: In the case of multiple image processing tasks to be processed, obtain the processing priority of each image processing task to be processed. It should be noted that the image processing task to be processed refers to a complete image processing task that has entered the task queue of the image processing system but has not yet started to be processed.

[0063] Step S200: Based on the processing priority of each image processing task to be processed, determine the target image processing task from each image processing task to be processed in descending order of priority, and perform the step of splitting the target image processing task into multiple sub-tasks. It should be noted that the target image processing task is the image processing task with the highest processing priority selected from all the image processing tasks to be processed.

[0064] Step S300: After the target image processing task is completed, return to the step of determining the target image processing task from the image processing tasks to be processed in descending order of priority according to the processing priority of each image processing task to be processed, until all image processing tasks to be processed are completed.

[0065] This embodiment, by defining the scenario where there are multiple image processing tasks to be processed in the image processing system, will process each image processing task in sequence according to its processing priority, so as to ensure that tasks with high processing priority can be processed in advance.

[0066] Based on the first, second, and / or third embodiments described above, a fourth embodiment of the image processing method of this application is proposed. In the fourth embodiment, the image processing method may further include steps S400-S500: Step S400: Monitor whether there are any abnormalities during the process of processing the target image; It should be noted that when monitoring whether there are any abnormalities in the process of processing the target image, the communication counter of the application (APP) or process in the multi-core chip can be monitored and timed. If a timeout is found, it can be determined that there is an abnormality in the process of processing the target image.

[0067] Step S500: If yes, obtain abnormal information during the processing of the target image processing task and report the abnormal information.

[0068] It should be noted that the exception information may include, but is not limited to, exception occurrence timestamp, exception type, error code, exception occurrence location, etc., and this embodiment does not impose specific limitations on it.

[0069] This embodiment monitors for anomalies in the image processing system during the processing of the target image processing task. If an anomaly is found, the relevant anomaly information will be reported in a timely manner, so that relevant personnel or equipment can handle the anomaly in the image processing system in a timely manner, thereby ensuring the stability and efficiency of image processing.

[0070] For example, to help understand the implementation principle of the image processing method formed by combining the above embodiments when applied to a vehicle, please refer to... Figure 5 , specifically: Step A: The four-channel surround view video stream outputs a video stream image with a certain resolution and frame rate. After serial deserialization by the serial deserialization module and low-level configuration processing, it is output to the video stream access interface (i.e., the input interface in the figure) corresponding to the multi-core chip. Step B: After the image stream enters the chip, it first goes to the image signal processor (ISP) for image denoising, color enhancement, automatic exposure and other image coloring processes to ensure the image display quality under different lighting scenarios; Step C: After the image signal processor finishes processing, it is sent to the image preprocessing module for adaptation of parameters such as resolution, frame rate, and image format. After adaptation, it is split into two video streams for processing. Step D: Of the two video streams, one is sent to the graphics processor for tasks such as image stitching and rendering, while the other is sent to the perception module after being adapted (downsampled) by the image pyramid, so that it can enter the corresponding convolutional neural network for convolutional processing to extract people, vehicles, obstacles or other targets in the video stream images. In step E, after the perception module extracts the relevant information of the target object (such as target type, horizontal and vertical distance), it will provide it to the backend planning and control module for continuous vehicle control in order to complete a complete set of autonomous driving related tasks.

[0071] The different image processing / object recognition tasks mentioned above are completed by different core processors. For example, steps A to C can be completed by a central processing unit; image stitching and rendering in step D can be completed by a graphics processing unit; image acceleration processing and deep learning deployment of convolutional neural networks in step D can be completed by an MMA (Matrix Multiply-Accumulate) processor and a neural network processor, respectively; in step E, during the process of controlling the vehicle using the target object, a real-time processor can be used to receive and forward relevant signals.

[0072] It should be noted that this example is only for the purpose of assisting in understanding this application and does not constitute a limitation on the image processing method of this application. Other technical solutions based on this concept are all within the protection scope of this application.

[0073] This application also provides an image processing system, which may include a multi-core chip. The core processors in the multi-core chip include at least two of the following: a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor. The multi-core chip also has a shared memory space. The image processing system is used to implement the steps of the image processing method described in the above embodiments.

[0074] In one feasible implementation, the core processors communicate via IPC (Inter-Process Communication), the processes within each core processor communicate via COOL (Communication Object-Oriented Layer), and the threads communicate via DDK (Device Development Kit). This rational deployment of communication mechanisms between core processors, processes, and threads reduces the complexity of the image processing system and the overall development cost.

[0075] Based on this, taking the core processors, including the central processing unit, graphics processing unit, neural network processor, and real-time processor, as examples, the deployment tasks and signal flow interaction principles of each core processor can be referred to... Figure 6 , specifically: The real-time processor focuses on real-time control and vision underlying software. It may include CP Autosar (Autosar Classic Platform), which is a vehicle control software framework based on the Autosar standard and is responsible for real-time vehicle control logic. It may also include a vision graphics basic software layer to provide basic software support for underlying graphics and images for vision tasks.

[0076] The central processing unit (CPU) is responsible for task scheduling and system-level management. It may include a functional application layer, which can be configured with a predictive scheduling mechanism module. It may also include middleware (which enables communication and resource management between modules) and perform underlying configuration (responsible for configuring underlying hardware and software parameters).

[0077] The matrix multiplication accelerator focuses on hardware acceleration of visual and SLAM (Simultaneous Localization and Mapping) algorithms. It can perform image link deployment and includes optical flow / visual SLAM mapping algorithms.

[0078] Neural network processors focus on accelerating inference using deep learning algorithms. By running deep learning algorithms, they can quickly extract target objects (such as people, cars, obstacles, etc.) from images.

[0079] The graphics processing unit (GPU) is responsible for image rendering and visual presentation. By running the image rendering engine, it can render and output the processed images in an intuitive way.

[0080] The image processing system provided in this application, employing the image processing method described in the above embodiments, can improve image processing efficiency and ensure real-time image processing. Compared with the prior art, the beneficial effects of the image processing system provided in this application are the same as those of the image processing method described in the above embodiments, and other technical features of this image processing system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0081] This application also provides a vehicle, which includes a vehicle body and the image processing system described above, wherein the image processing system is disposed on the vehicle body.

[0082] The vehicle provided in this application embodiment, employing the image processing system described above, can improve image processing efficiency to ensure real-time image processing. Compared with the prior art, the beneficial effects of the vehicle provided in this application embodiment are the same as those of the image processing system provided in the above embodiments, and will not be repeated here.

[0083] This application also provides a computer-readable storage medium storing a computer program that can run on a processor, the computer program being used to execute the image processing method described above.

[0084] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0085] The aforementioned computer-readable storage medium may be included in the image processing system or may exist independently without being assembled into the image processing system.

[0086] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the image processing system, cause the image processing system to: divide the target image processing task into multiple subtasks; determine the target core processor corresponding to each subtask in each core processor based on the inherent characteristic information of each core processor in the multi-core chip, and allocate each subtask to its corresponding target core processor; and allow each core processor to interact with data based on the shared memory space set up by the multi-core chip to execute each subtask in parallel; wherein each core processor includes at least two of the following: a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor.

[0087] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0089] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0090] The computer-readable storage medium provided in this application embodiment stores computer-readable program instructions for executing the above-described image processing method, which can improve image processing efficiency and ensure the real-time performance of image processing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the image processing method provided in the above embodiments, and will not be repeated here.

[0091] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image processing method described above.

[0092] The computer program product provided in this application can improve image processing efficiency to ensure real-time image processing. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the image processing method provided in the above embodiments, and will not be repeated here.

[0093] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Break down the target image processing task into multiple sub-tasks; Based on the inherent characteristics of each core processor in the multi-core chip, the target core processor corresponding to each subtask in each core processor is determined, and each subtask is assigned to its corresponding target core processor. Each of the core processors interacts with data based on the shared memory space set up by the multi-core chip in order to execute each of the subtasks in parallel; Each of the core processors includes at least two of the following: a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor.

2. The method as described in claim 1, characterized in that, The step of determining the target core processor corresponding to each subtask in each core processor based on the inherent characteristic information of each core processor in the multi-core chip includes: For any of the core processors, the target task type corresponding to the functional specialization of the core processor is determined based on the inherent characteristic information of the core processor. The core processor is designated as the target core processor for each of the subtasks whose task type is the target task type.

3. The method as described in claim 2, characterized in that, The target task type is either logic control type, image rendering type, image acceleration processing type, or system status management type. The step of designating the core processor as the target core processor for the subtask whose task type is the target task type in each of the subtasks includes: The central processing unit is used as the target core processor for the subtask whose task type is the logic control type. And / or, the graphics processor is used as the target core processor for the subtask whose task type is the image rendering type in each of the subtasks; And / or, the neural network processor is used as the target core processor for the subtask whose task type is the image acceleration processing type in each of the subtasks; And / or, the real-time processor is used as the target core processor for the subtask whose task type is the system state management type in each of the subtasks.

4. The method as described in claim 1, characterized in that, Before the step of assigning each of the subtasks to its corresponding target core processor, the method further includes: Obtain historical processing latency data and load change trends for each of the core processors; Based on the historical processing latency data and load change trends of each core processor, predict the future computing power requirements of each core processor; The proportion of tasks allocated to each core processor is adjusted based on the future computing power requirements of each core processor.

5. The method as described in claim 4, characterized in that, The step of adjusting the proportion of tasks allocated to each of the core processors based on the future computing power demand of each core processor includes: For any of the core processors, the target load state that the core processor is ready to enter is determined based on the future computing power requirements of the core processor. If the target load state is a high load state, then reduce the proportion of tasks allocated to the core processor; If the target load state is low load, then increase the proportion of tasks allocated to the core processor.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: In the case of multiple image processing tasks to be processed, the processing priority of each image processing task to be processed is obtained; Based on the processing priority of each of the image processing tasks to be processed, the target image processing task is determined from each of the image processing tasks to be processed in descending order of priority, and the step of splitting the target image processing task into multiple sub-tasks is executed. After the target image processing task is completed, the process returns to the step of determining the target image processing task from the image processing tasks to be processed in descending order of priority, until all the image processing tasks to be processed are completed.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Monitor whether any abnormalities occur during the processing of the target image processing task; If so, obtain abnormal information during the processing of the target image processing task and report the abnormal information.

8. An image processing system, characterized in that, The image processing system includes a multi-core chip, wherein each core processor in the multi-core chip includes at least two of a central processing unit, a graphics processing unit, a neural network processor, and a real-time processor, and the multi-core chip is provided with a shared memory space; the image processing system is used to implement the steps of the image processing method as described in any one of claims 1 to 7.

9. The image processing system as described in claim 8, characterized in that, Each of the core processors communicates via IPC, and each process within each core processor communicates via COOL, while each thread communicates via DDK.

10. A vehicle, characterized in that, The vehicle includes a vehicle body and an image processing system as described in claim 8 or 9, wherein the image processing system is disposed on the vehicle body.

Images