Image processing method, device, chip, controller and vehicle

Abstract

The application discloses an image processing method and device, a chip, a controller and a vehicle, and belongs to the technical field of images. The method comprises the following steps: receiving a plurality of first storage messages; when the image identifier carried in the first storage message indicates other images, reading the other images in sequence according to the storage position carried in the first storage message according to a preset reading strategy, and performing image processing on the other images until a second storage message is received; the image identifier carried in the second storage message indicates a front view image with the highest priority; reading the front view image according to the storage position of the front view image carried in the second storage message, and performing image processing on the front view image; after the processing of the front view image is completed, the other images that have not been processed are continuously read, and the processing of the other images that have not been processed is completed before a plurality of first storage messages in a next period are received. The processing priority of the front view image is ensured, the processing delay is reduced, and the image processing efficiency is improved.

Description

Image processing methods, devices, chips, controllers, and vehicles Technical Field

[0001] This application relates to the field of image technology, and in particular to an image processing method, device, chip, controller and vehicle. Background Technology

[0002] In driving scenarios, multiple image sensors are needed to acquire images of the vehicle from different perspectives, such as front view, rear view, surround view, and side view. These sensors transmit their acquired images via a video input (VI) module to an image processing chip. The chip then processes each image, and the processed images can be used to display real-time footage or for route planning.

[0003] Currently, considering the chip's hardware resources, there is an upper limit to the number of images a chip can process at any given time. The chip uses a first-in, first-out (FIFO) strategy to process the transmitted images sequentially until all images acquired by the image sensors in the current cycle have been processed.

[0004] However, different images have varying degrees of importance for subsequent screen display or route planning. Generally, the forward-view image is more important than other images from other viewpoints. This first-in-first-out (FIFO) approach, which processes transmitted images sequentially, can result in longer processing times for the forward-view image, reducing image processing efficiency. Summary of the Invention

[0005] This application provides an image processing method, device, chip, controller, and vehicle, which reduces the latency of front-view image processing and improves image processing efficiency. The technical solution is as follows:

[0006] Firstly, an image processing method is provided, comprising: receiving multiple first storage messages; the first storage messages indicating that the transmission of images acquired by an image sensor has been completed; when an image identifier carried in the first storage message indicates other images, other images are sequentially read according to a preset reading strategy based on the storage location carried in the first storage message, and image processing is performed on the other images until a second storage message is received; wherein, the image identifier carried in the second storage message indicates the front view image with the highest priority; this scheme performs image processing on the received other images according to the preset reading strategy during the time period when the front view image has not been transmitted, and improves image processing efficiency by utilizing limited time and hardware resources to process other images first without affecting the processing delay of the front view image. According to the storage location of the front view image carried in the second storage message, the front view image is read and image processing is performed on the front view image; after the processing of the front view image is completed, the unprocessed other images are sequentially read according to the preset reading strategy, and the processing of the unprocessed other images is completed before receiving multiple first storage messages in the next cycle. This scheme stops reading and processing other images after the front view image has been transmitted, and performs image processing on the received front view image. After the front view image is processed, other images that have not yet been processed are processed according to the preset reading strategy. This ensures the processing priority of the front view image, reduces the processing latency of the front view image, and improves the image processing efficiency.

[0007] In some embodiments, the preset reading strategy is a first-in-first-out strategy, or it is obtained by sorting according to the preset level corresponding to each image identifier, thereby improving the diversity of preset reading strategies.

[0008] In some embodiments, according to the storage location carried by the first storage message, other images are sequentially read according to a preset reading strategy, and image processing is performed on the other images until a second storage message is received. This includes: according to the storage locations carried by N first storage messages, N other images are sequentially read according to a preset reading strategy, and image processing is performed on the N other images simultaneously until a second storage message is received. In this solution, the chip can perform image processing on N images simultaneously. Based on this, for one processing operation, the chip can read N other images and assign them to N image processing sub-modules in the chip. Each image processing sub-module processes one other image, thus improving image processing efficiency.

[0009] Where N is a positive integer greater than 1, representing the number of images processed simultaneously. N is determined based on the image processing rate, the processing time of a single other image, the processing time of the front-view image, and the frame rate of the image sensor. The setting of N comprehensively considers the priority processing of the front-view image and the utilization rate of the chip's hardware resources, ensuring that the sum of the processing times of multiple other images is greater than the difference between the transmission time of the front-view image and the transmission time of a single other image, and that the sum of the processing times of multiple other images and the front-view image is less than or equal to the reciprocal of the frame rate. This ensures the priority processing of the front-view image while improving the utilization rate of hardware resources.

[0010] In some embodiments, the number of second storage messages is M, where M is a positive integer greater than 1 and less than or equal to N. Based on the storage location of the front view image carried by the second storage message, the front view image is read and image processing is performed on it. This includes: according to the storage location of the front view image carried by each of the M second storage messages, M front view images are read sequentially according to a first-in-first-out (FIFO) strategy, and image processing is performed on the M front view images simultaneously until all M front view images have been processed. In this solution, the chip can perform image processing on N images simultaneously. Based on this, for one processing operation, the chip can read M front view images and allocate them to M image processing sub-modules within the chip. Each image processing sub-module processes one front view image, thus processing M front view images in one operation, thereby improving image processing efficiency.

[0011] In some embodiments, the number of second stored messages is M, where M is a positive integer greater than N;

[0012] Based on the storage location of the front view image carried in the second storage message, the front view image is read and image processing is performed on it. This includes: according to the storage location of the front view image carried in each of the M second storage messages, N front view images are read sequentially according to a first-in-first-out (FIFO) strategy, and image processing is performed on the N front view images simultaneously until all M front view images have been processed. In this scheme, the chip can perform image processing on N images simultaneously. Based on this, for one processing operation, the chip can read N front view images and allocate them to N image processing sub-modules in the chip. Each image processing sub-module processes one front view image, and so on, until all M front view images have been processed. This improves image processing efficiency.

[0013] In some embodiments, the plurality of other images include at least one of the following: rear view image, surround view image, side view image, and interior view image in a driving scene; the M front view images include a telephoto image and a wide-angle image from a forward-looking perspective. In practical applications, different image combination methods can all employ the image processing method provided in the embodiments of this application, increasing the diversity of application scenarios.

[0014] In some embodiments, the image processing method further includes: receiving a second storage message; the image identifier carried in the second storage message indicating the highest priority front view image; reading the front view image according to the storage location of the front view image carried in the second storage message, and performing image processing on the front view image; after the front view image processing is completed, sequentially reading multiple other images according to the storage locations carried in multiple received first storage messages and following a preset reading strategy, and completing the processing of multiple other images before receiving the second storage message of the next cycle; wherein, the image identifier carried in the first storage message indicates other images. For application scenarios where the front view image is received first, the chip can directly perform image processing on the front view image first, and then sequentially read and process other images after the front view image processing is completed. By appropriately setting the chip's processing rate and the upper limit of the number of images the chip can process simultaneously, the processing of multiple other images can be completed before receiving the second storage message of the next cycle, that is, the processing of the front view image and multiple other images in the current cycle can be completed. Thus, when the second storage message of the next cycle is received, processing of the front view image transmitted in the next cycle can begin promptly, reducing image processing latency.

[0015] In a second aspect, a chip is provided, the chip including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the methods described in the first aspect and any embodiment of the first aspect.

[0016] Thirdly, an image processing device is provided, comprising a video input module and a chip as described in the second aspect; the video input module is configured to receive other images transmitted by a plurality of first image sensors, and to receive a front view image transmitted by a second image sensor; generate a first storage message after each of the other images has been transmitted, and generate a second storage message after the front view image has been transmitted; the plurality of first image sensors are configured to acquire at least one of a rear view image, a surround view image, a side view image, and an interior view image in a driving scene, respectively, and the second image sensor is configured to acquire a front view image in a driving scene; the chip is configured to receive multiple A first storage message is received; according to the storage location carried in the first storage message, the other images are read sequentially according to a preset reading strategy, and image processing is performed on the other images until a second storage message is received; wherein, the image identifier carried in the second storage message indicates the front view image with the highest priority; according to the storage location of the front view image carried in the second storage message, the front view image is read, and image processing is performed on the front view image; after the front view image is processed, the other unprocessed images are read sequentially according to the preset reading strategy, and the processing of the other unprocessed images is completed before receiving multiple first storage messages in the next cycle.

[0017] Fourthly, a controller is provided, the controller comprising the image processing apparatus as described in the third aspect above.

[0018] Fifthly, a vehicle is provided, the vehicle including a controller as described in the fourth aspect above.

[0019] In a sixth aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described in the first aspect and any embodiment of the first aspect.

[0020] In a seventh aspect, a computer program product including instructions is provided, which, when run on a computer, causes the computer to perform the methods described in the first aspect and any embodiment of the first aspect.

[0021] The technical effects achieved by the second, third, fourth, fifth, sixth, and seventh aspects described above are similar to the technical effects achieved by the corresponding technical means in the first aspect and any embodiment of the first aspect, and will not be repeated here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0024] Figure 2 is a schematic diagram of the structure of another image processing device provided in an embodiment of this application;

[0025] Figure 3 is a flowchart of an image processing method provided in an embodiment of this application;

[0026] Figure 4 is a flowchart of another image processing method provided in an embodiment of this application;

[0027] Figure 5 is a schematic diagram of an image processing timing provided in an embodiment of this application;

[0028] Figure 6 is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0031] Before providing a detailed explanation of the embodiments of this application, the application scenarios and related technologies of the embodiments of this application will be described first.

[0032] The image processing method provided in this application is applied to driving scenarios, such as real-time image display, route planning during autonomous driving, forward collision warning, lane departure warning, pedestrian detection, 360-degree panoramic view, and reversing image. These scenarios have requirements regarding image processing latency. In particular, for the Automatic Emergency Braking (AEB) system in driving scenarios, the latency of the forward view image is critical, and it is necessary to minimize the delay of the forward view image.

[0033] In related solutions, for images transmitted from the video input module, the chip can also employ a priority scheduling method, processing images sequentially according to their priority from high to low. Since the front view image has a higher priority than other images, the chip waits for the front view image to complete its transmission before processing it, and then processes the other images. While this method reduces the processing latency of the front view image, it results in a delay in processing other images, reducing the real-time performance of image processing for scenarios requiring subsequent generation of 3D visual animations or reversing maneuvers.

[0034] The image processing method provided in this application is applied to an image processing device, which can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the image processing device can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of image processing device.

[0035] As shown in Figure 1, Figure 1 is a structural schematic diagram of an image processing device 100 provided in an embodiment of this application. The image processing device 100 includes a video input module 10 and a chip 20. The video input module 10 is used to receive other images transmitted by multiple first image sensors, and to receive a front view image transmitted by a second image sensor. After each other image is transmitted, it generates a first storage message, and after the front view image is transmitted, it generates a second storage message. The multiple first image sensors are used to collect at least one of the following in a driving scene: a rear view image, a surround view image, a side view image, and an interior view image. The second image sensor is used to collect the front view image in a driving scene. Image; Chip 20, used to receive multiple first storage messages; according to the storage location carried by the first storage message, sequentially read other images according to a preset reading strategy, and perform image processing on the other images until a second storage message is received; wherein, the image identifier carried by the second storage message indicates the front view image with the highest priority; according to the storage location of the front view image carried by the second storage message, read the front view image and perform image processing on the front view image; after the front view image processing is completed, continue to sequentially read other unprocessed images according to the preset reading strategy, and complete the processing of other unprocessed images before receiving multiple first storage messages in the next cycle.

[0036] In this embodiment, chip 20 is used for image processing. Chip 20 may be called an image signal processor (IPS), image processor, or image processing module, etc. This embodiment does not limit the scope of the application.

[0037] In this embodiment, multiple image sensors are used, each positioned at a different viewpoint of the vehicle to capture images from those perspectives. All image sensors operate at the same frame rate and begin capturing images at the same time. After each sensor completes its image capture, multiple images can be simultaneously transmitted to the video input module 10. Image transmission from the image sensors to the video input module 10 is performed frame by frame. Each time the video input module 10 receives a frame, it requests an image buffer from the storage module and stores that frame in the image buffer. Then, it instructs the chip 20 to read the frame from the image buffer in the storage module for image processing. This process continues, with the video input module 10 storing each frame in its corresponding image buffer, and the chip 20 sequentially reading each frame from the image buffer in the storage module.

[0038] The video input module 10 receives images (other images or front view images) transmitted from multiple image sensors (including multiple first image sensors and second image sensors). The transmission duration of each other image can be the same or different. Here, in this embodiment, the minimum transmission duration among multiple other images is taken as the transmission duration of a single other image. The transmission duration of the front view image can be greater than the transmission duration of a single other image, or the transmission duration of the front view image can be equal to or less than the transmission duration of a single other image.

[0039] In scenarios where the transmission time of the front view image is longer than the transmission time of any other image, for the current cycle, the video input module 10 first stores the other images that have been transmitted in the storage module, then stores the front view image that has been transmitted in the storage module, and then enters the next cycle.

[0040] For scenarios where the transmission duration of the front view image is less than or equal to the transmission duration of a single other image, for the current cycle, the video input module 10 first stores the transmitted front view image in the storage module, then stores the transmitted other images in the storage module, and then enters the next cycle.

[0041] After storing an image, the video input module 10 generates storage information including image-related information (e.g., the image sensor from which the image originated, and the image's storage location), and sends this storage information to the chip 20. For example, the video input module 10 generates a first storage message after each other image has been transmitted, and a second storage message after the previous image has been transmitted. If there are multiple previous images, the video input module 10 generates a second storage message after each previous image has been transmitted.

[0042] In this embodiment, the plurality of first image sensors include at least one of an interior view camera, a rear view camera, a side view camera, and a surround view camera. The interior view camera is used to acquire images of the driver's location or the locations of other occupants in the vehicle, referred to as an interior view image. The interior view image may include DMS images from a Driver Monitoring System (DMS) and / or OMS images from an Occupant Monitoring System (OMS). The rear view camera is used to acquire rear view images, the side view camera is used to acquire side view images, and the surround view camera is used to acquire surround view images. In scenarios where the transmission time of the front view image is greater than the transmission time of any other individual image, the transmission time of the rear view image, surround view image, side view image, or interior view image is less than that of the front view image. The transmission times of the rear view image, surround view image, side view image, and interior view image may be the same or different.

[0043] In this embodiment, the second image sensor is a forward-facing camera used to capture forward-facing images in a driving scenario. A vehicle may have multiple forward-facing cameras, such as a main forward-facing camera, a wide-angle forward-facing camera, and a telephoto forward-facing camera, which can be used to capture images at different monitoring distances. The main forward-facing camera captures the main image within the main field of view, the wide-angle forward-facing camera captures the W image (i.e., wide-angle image) within a wide field of view, and the telephoto forward-facing camera captures the T image (i.e., telephoto image) within a narrow field of view.

[0044] It should be noted that there can be multiple side-view cameras and surround-view cameras on a vehicle. For example, a vehicle may integrate an interior camera, a rear-view camera, four side-view cameras, four surround-view cameras, and two front-view cameras. This application does not limit the number of such cameras.

[0045] After the image acquired by the image sensor is transmitted, the video input module 10 generates storage information (first storage information or second storage information) and sends it to the chip 20. In scenarios where the transmission time of the front-view image is longer than the transmission time of any other image, since the first storage information is generated and sent before the second storage information, the chip 20 receives the first storage information first. Based on the image identifier carried in the first storage information, the chip 20 can determine that the image originates from the first image sensor, which has a lower priority than images acquired by the second image sensor. However, since the second storage information indicating that the image originates from the second image sensor has not yet been received, the chip 20 can sequentially read other images according to the storage location carried in the first storage message and perform image processing on them, until the second storage information is received, at which point it stops reading and processing other images.

[0046] After receiving the second storage information, chip 20 reads the front view image according to the storage location of the front view image carried in the second storage message and performs image processing on the front view image. This ensures the processing priority of the front view image, reduces the processing latency, and improves image processing efficiency. After processing the front view image, it continues to read other unprocessed images sequentially according to the storage location carried in the unprocessed first storage messages and a preset reading strategy, making full use of limited time and hardware resources and improving the real-time performance of image processing. Furthermore, it completes the processing of other unprocessed images before receiving multiple first storage messages in the next cycle. Thus, when multiple first storage messages are received in the next cycle, other images transmitted in the next cycle can be processed promptly, reducing image processing latency.

[0047] It should be noted that if there are multiple front view images, all front view images in the current cycle must be processed before other images can be processed.

[0048] In scenarios where the transmission time of the front view image is less than or equal to the transmission time of a single other image, since the second storage information is generated and sent before the first storage information, chip 20 receives the second storage information first. Based on the image identifier carried in the second storage information, chip 20 can determine that the image originates from the second image sensor, which has the highest priority. Therefore, chip 20 directly reads the front view image according to the storage location carried in the second storage message and performs image processing on the front view image. Even if the first storage information indicating that the image originates from the first image sensor is received before the front view image processing is completed, the reading and processing of the front view image will not stop. After the front view image processing is completed, chip 20 sequentially reads multiple other images according to the storage location carried in the first storage message and follows a preset reading strategy, and performs image processing on each of the other images.

[0049] The storage module in this application embodiment can be a Double Data Rate (DDR) synchronous dynamic random access memory.

[0050] Based on Figure 1 above, and as shown in Figure 2, Figure 2 is a structural schematic diagram of another image processing device 100 provided in an embodiment of this application. Below, an exemplary application of an embodiment of this application in a practical application scenario will be described with reference to Figure 2.

[0051] The image processing device 100 in Figure 2 further includes a video input management module 30, an image processing management module 40, a parameter configuration module 50, and a storage module 60. The parameter configuration module 50 configures the storage logic of the video input module 10, and the video input management module 30 controls the video input module 10 to perform image storage based on this configuration. Under the control of the video input management module 30, when the video input module 10 receives each frame of image, it requests an image buffer from the storage module 60 to store each frame of image in the image buffer. After the video input module 10 completes storage, it informs the video input management module 30. The video input management module 30 generates storage information based on the image identifier and storage location, and sends the storage information to the image processing management module 40 via the parameter configuration module 50 (Param Config, Param CFG). Alternatively, the video input management module 30 and the image processing management module 40 are communicatively connected. After generating the storage information, the video input management module 30 directly sends the storage information to the image processing management module 40. This embodiment of the application does not limit the communication method.

[0052] The parameter configuration module 50 is used to configure the processing logic of the chip 20. The image processing management module 40 controls the chip 20 to implement image processing based on this configuration. The image processing management module 40 can be a command queue (CDQ). The CDQ, based on the configuration of the parameter configuration module 50, enables the hardware mechanism of the chip 20 to implement the image processing method. The CDQ can also be represented as a CMD queue. For example, the parameter configuration module 50 can configure parameters of the chip 20, such as preset reading strategies for other images and a highest priority processing strategy for the previous image. The parameter configuration module 50 communicates with the video input management module 30, receiving storage information sent by the video input management module 30. Alternatively, the image processing management module 40 communicates with the video input management module 30, receiving storage information sent by the video input management module 30.

[0053] In Figure 2 above, under the control of the parameter configuration module 50, the chip 20 performs image processing according to different strategies based on the image identifiers carried by the stored information. That is, when the image identifiers carried by multiple stored messages all indicate other images, the chip 20 reads other images sequentially according to the storage location carried by the stored messages and the preset reading strategy, and performs image processing on the other images. When the image identifiers carried by multiple stored messages include image identifiers indicating other images and image identifiers indicating the front view image, the chip 20 prioritizes reading the front view image according to the storage location carried by the storage message corresponding to the front view image, and performs image processing on the front view image.

[0054] In this embodiment, when there are multiple image sensor inputs, the video input module 10 is responsible for receiving the images output by the multiple image sensors and outputting them to the image buffer respectively, thereby storing multiple images (including one or more front view images and multiple other images). The chip 20 needs to process the images acquired by each of the multiple image sensors, sequentially reading each frame from the image buffer of the video input module 10, and obtaining the target image after image processing.

[0055] It should be noted that chip 20 can process 1 frame of image at the same time. Of course, chip 20 can also process 2 or more frames of image at the same time. The upper limit of the number of images that chip 20 can process can be appropriately set by those skilled in the art according to the actual situation. This application embodiment does not limit this.

[0056] In this embodiment, during the current cycle, multiple first image sensors each acquire other images and transmit the acquired images to the video input module 10; simultaneously, a second image sensor acquires a front view image and transmits the acquired front view image to the video input module 10. The multiple sensors (including multiple first and second image sensors) have the same frame rate, and the multiple sensors can start acquiring images substantially simultaneously. After acquisition, the multiple images begin transmission substantially simultaneously. For each other image, the video input module 10 generates a first storage message after the other images have been transmitted; for the front view image, the video input module 10 generates a second storage message after the front view image has been transmitted.

[0057] For scenarios where the transmission time of the front view image is longer than the transmission time of any other individual image, chip 20 first receives multiple first storage messages sent by video input module 10, identifies the image identifiers carried in the first storage messages indicating other images, and sequentially reads other images according to the storage locations carried in the first storage messages based on a preset reading strategy, performing image processing on these other images until a second storage message is received. When the image identifier carried in the second storage message indicates the front view image with the highest priority, reading and image processing of other images ceases. The front view image is read based on the storage location carried in the second storage message, and image processing is performed on the front view image. After the front view image processing is completed, the unprocessed other images continue to be read, and the processing of the unprocessed other images is completed before receiving multiple first storage messages in the next cycle. This solution ensures the processing priority of the front view image, reduces the processing latency of the front view image, and improves image processing efficiency.

[0058] For scenarios where the transmission time of the foreground image is less than or equal to the transmission time of a single other image, chip 20 first receives the second storage message sent by video input module 10, identifies the image identifier carried in the second storage message indicating the foreground image with the highest priority, reads the foreground image according to the storage location carried in the second storage message, and performs image processing on the foreground image until the processing of the foreground image is complete. Then, based on the storage locations carried in multiple received first storage messages, multiple other images are read sequentially, and the processing of multiple other images is completed before receiving the second storage message of the next cycle. This scheme ensures the processing priority of the foreground image, reduces the processing latency of the foreground image, and improves image processing efficiency.

[0059] In some embodiments, the video input module 10 is further configured to, after receiving other images transmitted by the multiple first image sensors respectively, store the multiple other images in multiple first preset image storage spaces; and generate each first storage message according to each image identifier used to indicate the source of the image and each storage location used to indicate the first preset image storage space.

[0060] For each other image, the first image sensor transmits the other image to the video input module 10. The video input module 10 requests a first preset image storage space from the storage module 60 and stores the other image in the first preset image storage space. After storage is completed, a first storage message is generated based on the image identifier of the image sensor from which the other image originated and the location of the first preset image storage space. This process is repeated for each other image, thereby achieving the storage of multiple other images and generating multiple first storage messages.

[0061] In some embodiments, the video input module 10 is further configured to store the front view image in a second preset image storage space after receiving the front view image transmitted by the second image sensor; and generate a second storage message based on the image identifier used to indicate the image source and the storage location used to indicate the second preset image storage space.

[0062] For the front view image, the second image sensor transmits the front view image to the video input module 10. The video input module 10 requests a second preset image storage space from the storage module 60 and stores the front view image in the second preset image storage space. After storage is completed, a second storage message is generated based on the image identifier of the image sensor from which the front view image originated and the location of the second preset image storage space, thus realizing the storage of the front view image and generating the second storage message.

[0063] In some embodiments, the number of second image sensors is M; the video input module 10 is further configured to receive the front view images transmitted by each of the M second image sensors, and generate M second storage messages after the transmission of the M front view images is completed.

[0064] When there are multiple front view images, for each front view image, the video input module 10 requests a second preset image storage space from the storage module 60 and stores the front view image in the second preset image storage space. After storage is completed, a second storage message is generated based on the image identifier of the image sensor from which the front view image originated and the location of the second preset image storage space. This process is repeated for each front view image, thereby achieving the storage of multiple front view images and generating multiple second storage messages.

[0065] Based on the image processing device 100 provided in FIG1, this application embodiment provides an image processing method, which can be executed by the chip 20 in FIG1. ​​As shown in FIG3, FIG3 is a flowchart of an image processing method provided in this application embodiment, which includes:

[0066] S101, Receive multiple first stored messages; the first stored messages are used to indicate that the image acquired by the image sensor has been transmitted.

[0067] In this embodiment, the storage message (first storage information or second storage message) is generated after the image acquired by the image sensor has been transmitted. The chip can receive multiple first storage messages almost simultaneously, or sequentially. In one approach, the first storage message carries an image identifier; after receiving the first storage information, the chip can directly identify the image as another image. In another approach, the first storage message is obtained by encapsulating the image identifier and storage location; after receiving the first storage information, the chip parses the first storage message to extract the image identifier, thereby identifying the image as another image.

[0068] S102. When the image identifier carried by the first storage message indicates other images, other images are read sequentially according to the storage location carried by the first storage message and a preset reading strategy is followed, and image processing is performed on the other images until the second storage message is received; wherein, the image identifier carried by the second storage message indicates the front view image with the highest priority.

[0069] In some embodiments, the plurality of other images include at least one of a rear view image, a surround view image, a side view image, and an interior view image in a driving scene.

[0070] In some embodiments, the preset reading strategy is a first-in-first-out strategy, or it is obtained by sorting according to the preset level corresponding to each image identifier.

[0071] The preset reading strategy can be a first-in, first-out (FIFO) strategy, which means that the reading order of multiple other images is determined according to the receiving order of multiple first stored messages. The other images corresponding to the second stored information received earlier are read first, and the other images corresponding to the second stored information received later are read last.

[0072] The preset reading strategy can also be obtained by sorting the preset levels corresponding to each image identifier, that is, pre-setting the reading order of multiple image identifiers. Those with higher preset levels are read first, and those with lower preset levels are read later. For example, the preset levels from highest to lowest can be set as: side view image, panoramic view image, rear view image, interior view image, or panoramic view image, side view image, interior view image, rear view image. For cases with multiple side view images and multiple panoramic view images, taking four as an example, the preset levels from highest to lowest can be set as: side view image 0, side view image 1, side view image 2, side view image 3, panoramic view image 0, panoramic view image 1, panoramic view image 2, panoramic view image 3, rear view image, interior view image; or side view image 0, panoramic view image 0, side view image 1, panoramic view image 1, side view image 2, panoramic view image 2, side view image 3, panoramic view image 3, rear view image, interior view image. The preset levels can be appropriately set by those skilled in the art according to actual circumstances, and this application embodiment does not limit this.

[0073] In scenarios where the transmission time of a front-view image exceeds the transmission time of any other individual image, the chip identifies the first storage message indicating other images and sequentially reads these images according to the storage location carried in the first storage message. Image processing is then performed on these other images until a second storage message carrying an identifier indicating the image with the highest priority is received, at which point reading and image processing of other images ceases.

[0074] Based on the above description of the preset reading strategy, in one approach, the chip receives the first storage message sequentially and reads other images sequentially according to a first-in, first-out (FIFO) strategy. In another approach, the chip can receive multiple first storage messages simultaneously; therefore, after receiving multiple first storage messages, it can read other images sequentially according to a preset reading order. Even when the first storage messages are received sequentially, it occurs within a very short time, thus allowing for the sequential reading of other images according to a preset reading order.

[0075] Image processing in this application refers to Image Signal Processing (ISP), including but not limited to image denoising, contrast enhancement, image correction, color space conversion, image compression, image enhancement, and image blurring. Image processing can also be called image effect processing, image quality processing, or image image processing, and this application does not limit the terminology used in this embodiment.

[0076] S103. Based on the storage location of the front view image carried in the second storage message, read the front view image and perform image processing on the front view image.

[0077] After receiving the second storage message, the chip identifies the image identifier carried in the message, which has the highest priority, and stops reading and processing any new images. For example, if another image is already being processed, the chip waits for that image to finish before starting to read and process the front view image. If the image processing of another image has been completed and the next reading process is about to begin, the chip directly reads and processes the front view image.

[0078] S104. After the processing of the front view image is completed, continue to read other unprocessed images in sequence according to the preset reading strategy, and complete the processing of other unprocessed images before receiving multiple first storage messages in the next cycle.

[0079] After processing the front view image, other unprocessed images are sequentially read according to the storage location carried by the unprocessed first storage message and a preset reading strategy. This makes full use of limited time and hardware resources, improving the real-time performance of image processing. By appropriately setting the chip's processing rate and the upper limit of the number of images the chip can process simultaneously, the processing of other unprocessed images is completed before receiving multiple first storage messages for the next cycle. In other words, the processing of multiple other images and the front view image in the current cycle is completed. Thus, when multiple first storage messages for the next cycle are received, other images transmitted in the next cycle can be processed promptly, reducing image processing latency.

[0080] According to the scheme provided in this application, multiple first storage messages are received. These first storage messages indicate that the image acquired by the image sensor has completed transmission. When the image identifier carried in the first storage message indicates another image, other images are sequentially read according to a preset reading strategy based on the storage location carried in the first storage message, and image processing is performed on these other images until a second storage message is received. The image identifier carried in the second storage message indicates the highest priority front-view image. This scheme processes other received images according to a preset reading strategy during the time period before the front-view image transmission is complete. Without affecting the processing latency of the front-view image, it utilizes limited time and hardware resources to process other images first, improving image processing efficiency. Based on the storage location of the front-view image carried in the second storage message, the front-view image is read and processed. After processing the front-view image, the unprocessed other images are read sequentially according to the preset reading strategy, and processing of the unprocessed other images is completed before receiving multiple first storage messages in the next cycle. After the front-view image transmission is complete, this scheme stops reading and processing other images and performs image processing on the received front-view image. After the front view image is processed, other images that have not yet been processed are processed according to the preset reading strategy. This ensures the processing priority of the front view image, reduces the processing latency of the front view image, and improves the image processing efficiency.

[0081] In some embodiments, for scenarios where the transmission time of the front view image is less than or equal to the transmission time of a single other image, the image processing method further includes the following steps: receiving a second storage message; the image identifier carried in the second storage message indicates the front view image with the highest priority; reading the front view image according to the storage location of the front view image carried in the second storage message, and performing image processing on the front view image; after the processing of the front view image is completed, reading multiple other images sequentially according to the storage locations carried in the multiple first storage messages received, following a preset reading strategy, and completing the processing of the multiple other images before receiving the second storage message of the next cycle; wherein, the image identifier carried in the first storage message indicates the other images.

[0082] In scenarios where the transmission time of a front view image is less than or equal to the transmission time of a single other image, the chip first receives the second storage message. After receiving the second storage message, the chip identifies the image identifier carried in the message, which has the highest priority. Therefore, it directly reads the front view image and performs image processing on it until processing is complete. If there are multiple front view images, image processing continues until all images are processed. The first storage message can be received during front view image processing or when preparing to process the next front view image. Even if a first storage message is received and the chip identifies it as indicating another image, it will not stop reading and processing the front view images until all front view images have been processed.

[0083] After processing the front view image, the chip sequentially reads other images according to the storage location carried in the first storage message. It then performs image processing on these other images. By appropriately setting the chip's processing rate and the upper limit on the number of images the chip can process simultaneously, the processing of multiple other images can be completed before receiving the second storage message for the next cycle. In other words, the processing of the front view image and multiple other images in the current cycle is completed. Thus, upon receiving the second storage message for the next cycle, processing of the front view image transmitted in the next cycle can begin promptly, reducing image processing latency.

[0084] In some embodiments, the chip for executing the image processing method may have multiple image processing sub-modules to process multiple images simultaneously. For example, the chip may include two image processing sub-modules, which can read two images at the same time and process them separately in the two image processing sub-modules. Based on this, embodiments of this application also provide an image processing method, as shown in FIG4, which is a flowchart of another image processing method provided by embodiments of this application.

[0085] S201, Receive multiple first stored messages; the first stored messages are used to indicate that the image acquired by the image sensor has been transmitted.

[0086] In this example, S201 is the same as S101 in Figure 3 above. Its implementation method and the technical effect achieved can be found in the description of S101 above, and will not be repeated here.

[0087] S202. When the image identifier carried by the first storage message indicates other images, according to the storage locations carried by the N first storage messages, N other images are read sequentially according to a preset reading strategy, and image processing is performed on the N other images simultaneously until the second storage message is received.

[0088] Where N is a positive integer greater than 1, N represents the number of images processed simultaneously. N is determined based on the image processing rate, the processing time of a single other image, the processing time of the front view image, and the frame rate of the image sensor, such that the sum of the processing times of multiple other images is greater than the difference between the transmission time of the front view image and the transmission time of a single other image, and the sum of the processing times of multiple other images and the front view image is less than or equal to the reciprocal of the frame rate.

[0089] In this embodiment, the setting of N needs to comprehensively consider both the priority processing of the foreground image and the hardware resource utilization of the image processing submodule. Specifically, considering the hardware resource utilization of the image processing submodule, it is unnecessary to set up too many image processing submodules; it is sufficient to ensure that the sum of the processing times of multiple other images and the foreground image is less than or equal to the reciprocal of the frame rate, where the reciprocal of the frame rate is the time taken for the image sensor to acquire one frame. Considering the priority processing of the foreground image, reading and processing of the foreground image begins after its transmission is complete, and processing of other images continues after the foreground image processing is finished. That is, after the foreground image transmission is complete, other images have not yet been fully processed, and processing of other images begins after their transmission is complete. Therefore, it is necessary to ensure that the sum of the processing times of multiple other images is greater than the difference between the transmission time of the foreground image and the transmission time of a single other image.

[0090] Based on this, under the conditions that the sum of the processing times of multiple other images is greater than the difference between the transmission time of the front view image and the transmission time of a single other image, and the sum of the processing times of multiple other images and the front view image is less than or equal to the reciprocal of the frame rate, the specific value of N is determined according to the image processing rate of the image processing submodule, the processing time of a single other image, the processing time of the front view image, and the frame rate of the image sensor. In practical applications, when the processing times of a single other image and the processing time of the front view image are both fixed, the above conditions can be met by increasing the image processing rate and / or increasing the number of image processing submodules.

[0091] The chip can process N images simultaneously. Based on this, for a single processing run, the chip can read N other images and distribute them among N image processing sub-modules. Each sub-module processes one image, thus improving image processing efficiency. If N equals 2, the chip can process 2 images simultaneously. The chip reads 2 other images sequentially according to a preset reading strategy and processes each image simultaneously. This process continues until the second stored message is received.

[0092] For example, with N equal to 2, the preset reading strategy is: side view image 0, side view image 1, side view image 2, side view image 3, panoramic view image 0, panoramic view image 1, panoramic view image 2, panoramic view image 3, rear view image, and interior view image. The chip reads side view image 0 and side view image 1, and performs image processing on side view image 0 and side view image 1 respectively. Then, the chip reads side view image 2 and side view image 3, and performs image processing on side view image 2 and side view image 3 respectively, and so on, until the second storage message is received.

[0093] For example, with N equal to 2, the preset reading strategy is: side view image 0, surround view image 0, side view image 1, surround view image 1, side view image 2, surround view image 2, side view image 3, surround view image 3, rear view image, and interior view image. The chip reads side view image 0 and surround view image 0, and performs image processing on them respectively. Then, the chip reads side view image 1 and surround view image 1, and performs image processing on them respectively, and so on, until the second storage message is received.

[0094] If the number of second stored messages is M, then after S202, if M is a positive integer greater than 1 and less than or equal to N, S203 can be executed after S202; if M is a positive integer greater than N, S204 can be executed after S202.

[0095] In some embodiments, the M front view images include a telephoto image and a wide-angle image from the front view perspective; the M front view images may also include a main image from the front view perspective.

[0096] S203. When the number of second storage messages is M, where M is a positive integer greater than 1 and less than or equal to N, according to the storage location of the front view image carried by each of the M second storage messages, the M front view images are read sequentially according to the first-in-first-out strategy, and image processing is performed on the M front view images simultaneously until the processing of the M front view images is completed.

[0097] In this example, there are M front view images, and the chip can process N images simultaneously. The chip reads the M front view images sequentially according to a first-in-first-out strategy or randomly, and processes each of the M front view images simultaneously to complete the processing of the M front view images.

[0098] For example, taking N equal to 2 and M equal to 2 as an example, after receiving two second storage messages, the chip reads two front view images in sequence according to the first-in-first-out strategy or randomly, and performs image processing on the two front view images simultaneously, thereby completing the processing of the two front view images.

[0099] S204. When the number of second storage messages is M, where M is a positive integer greater than N, according to the storage location of the front view image carried by each of the M second storage messages, N front view images are read sequentially according to the first-in-first-out strategy, and image processing is performed on the N front view images simultaneously until the processing of the M front view images is completed.

[0100] In this example, there are M front view images, and the chip can process N images simultaneously. The chip reads N front view images sequentially according to a first-in-first-out (FIFO) strategy or randomly, and processes each of the N front view images simultaneously. Then, it reads N more front view images sequentially according to a FIFO strategy or randomly, and processes each of the N front view images simultaneously, until all M front view images have been processed.

[0101] For example, taking N=2 and M=3 as an example, after receiving three second storage messages, the chip reads two front view images sequentially according to a first-in-first-out strategy or randomly, and performs image processing on each of the two front view images simultaneously. Then it reads one more front view image and performs image processing on that image simultaneously, thus completing the processing of the three front view images.

[0102] S205. After processing the M front view images, continue to read the other unprocessed images in sequence according to the preset reading strategy, and complete the processing of the other unprocessed images before receiving multiple first storage messages in the next cycle.

[0103] Since the front view images have the highest priority, other images should only be processed after all the front view images have been processed.

[0104] For example, let's say the front view image includes the front view W image and the front view T image, and the preset reading strategy is side view image 0, circumferential view image 0, side view image 1, circumferential view image 1, side view image 2, circumferential view image 2, side view image 3, circumferential view image 3, rear view image, and interior view image. If the chip receives stored information about the front view W image and the front view T image while processing or processing circumferential view image 2 and circumferential view image 3, then the final image processing queue is: side view image 0, circumferential view image 0, side view image 1, circumferential view image 1, side view image 2, circumferential view image 2, side view image 3, circumferential view image 3, front view W image, front view T image, rear view image, and interior view image.

[0105] In this embodiment, images transmitted by multiple image sensors are acquired. These images include multiple front view images and multiple other images, with the front view images having the highest priority. The chip first receives stored information about the other images, but has not yet received stored information about the front view images. N other images are read and processed each time until the stored information about the front view images is received, at which point reading and processing of other images ceases. After receiving the stored information about the front view images, N front view images are read and processed each time until all front view images have been processed. Then, N unprocessed other images are read and processed again until all other images have been processed. This scheme utilizes limited time and hardware resources to process other images promptly before prioritizing the front view images, reducing the processing latency of the front view images and thus improving image processing efficiency.

[0106] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0107] Based on Figures 1-4 above, taking a 12-channel sensor access video input as an example, the image timing diagram for image transmission and image processing is shown in Figure 5. Figure 5 is a schematic diagram of an image processing timing diagram provided by an embodiment of this application. The multiple other images shown in Figure 5 include side view image 0, side view image 1, side view image 2, side view image 3, panoramic view image 0, panoramic view image 1, panoramic view image 2, panoramic view image 3, rear view image, and interior view image. The multiple front view images shown in Figure 5 include the front view T image and the front view W image (i.e., M equals 2). The chip shown in Figure 5 includes two image processing sub-modules (i.e., N equals 2), namely, image processing sub-module 1 and image processing sub-module 2. The following is a description.

[0108] Figure 5 shows that the image sensor operates at a frame rate of 30 frames per second (FPS), and the time taken to acquire one frame is approximately 33 ms. Multiple image sensors simultaneously begin transmitting their acquired images. It takes 20 ms for the two front-view sensors to transmit one frame of the front-view image (front-view W image or front-view T image), while the other 10 sensors take 10 ms to transmit one frame of another image. That is, the transmission time for other images is 10 ms, and the transmission time for the front-view image is 20 ms. Thus, one cycle is approximately 33 ms. The chip needs to process all images in the current cycle before the transmission of multiple other images in the next cycle is complete; that is, it needs to complete the processing of 12 images within 33 ms.

[0109] It should be noted that Figure 5 is only used as an example where the transmission time of each other image is 10ms. It is understood that in actual applications, the transmission time of each other image may be different, and this embodiment of the application does not limit this.

[0110] The CMD Queue corresponding to the Image Process executes the configuration logic of the parameter configuration module to perform image processing on each image. Within the same time period, the two image processing sub-modules in the Image Process can process images simultaneously. In other words, the Image Process can process two images at the same time.

[0111] After receiving the other images transmitted by the 10 sensors at 10ms, the parameter configuration module configures 10 other images for the Image Process, which then processes them in a first-in, first-out (FIFO) manner via the CMD Queue. Figure 5 shows the order in image processing queue 1 as side view image 0, side view image 1, side view image 2, side view image 3, and rear view image; and the order in image processing queue 2 as circumferential view image 0, circumferential view image 1, circumferential view image 2, circumferential view image 3, and interior view image.

[0112] During the processing of the other 10 frames, the transmission of the two front view images is completed. Figure 5 shows the front view images transmitted by the two sensors at 20ms. The parameter configuration module configures the two front view images to the Image Process, that is, it processes them according to priority mode through the CMD Queue, stopping the processing of other images and prioritizing the processing of the two front view images.

[0113] Taking a processing time of approximately 3ms for each other image as an example, when the two image processing submodules are processing the 4th frame image (side view image 3 and panoramic view image 3), they receive the front view images (front view W image and front view T image). Therefore, after the 4th frame image is processed, the front view images need to be processed, instead of the rear view images in image processing queue 1 and the interior view images in image processing queue 2. Thus, the final processing order of image processing submodule 1 should be side view image 0, side view image 1, side view image 2, side view image 3, front view W image, and rear view image; the final processing order of image processing submodule 2 should be panoramic view image 0, panoramic view image 1, panoramic view image 2, panoramic view image 3, front view T image, and interior view image.

[0114] After processing the current frame, the Image Processor's hardware scheduling mechanism prioritizes higher-priority images (i.e., the previous image) without software intervention, thus improving processing efficiency. This hardware scheduling mechanism refers to the hardware logic implemented by the CMD Queue based on the parameter configuration module.

[0115] In this embodiment, when processing images from multiple sensors, hardware resources can be effectively utilized, and a priority-based, first-in-first-out (FIFO) parallel scheduling strategy can be implemented using hardware mechanisms. That is, without affecting the processing latency of the preceding image, other images are processed first using limited time and hardware resources. The highest-priority preceding image is processed promptly. After the preceding image is processed, other unprocessed images are then processed, ensuring the processing priority of the preceding images, reducing processing latency, and improving image processing efficiency.

[0116] In this embodiment, the processing time for each other image is approximately 3ms, and the processing time for each front view image is approximately 13ms. The chip has two image processing sub-modules. The time it takes for the chip to process 12 frames of images in the current cycle is 3ms × 5 + 13ms = 28ms, which is less than 33ms and does not affect the real-time processing of images in the next cycle. Furthermore, it can promptly release the storage space of the current cycle, preventing images from being overwritten in the storage module, thus avoiding excessive storage space occupation and improving space resource utilization. If the chip has only one image processing sub-module, the time it takes to process 12 frames of images is 3ms × 10 + 13ms × 2 = 56ms, which is greater than 33ms and would affect the real-time processing of images in the next cycle. If the chip includes three image processing sub-modules, the time it takes to process 9 other images is 3ms × 3 = 9ms. While processing the last other image, the front view image is received, and processing of the front view image begins after processing the last other image. This approach fails to reflect the highest priority of the front-view images. After processing all images in the current cycle in advance, the chip remains idle for a considerable period, reducing the utilization rate of hardware resources.

[0117] Based on this, the embodiments of this application only require two image processing sub-modules. The two image processing sub-modules satisfy the following conditions: the total processing time of multiple other images is greater than the difference between the transmission time of the front view image and the transmission time of a single other image, and the total processing time of multiple other images and the front view image is less than or equal to the reciprocal of the frame rate, that is, satisfying 3ms×5>20ms-10ms and 3m×5+13ms<33ms.

[0118] Compared to priority-based scheduling, which processes images sequentially from highest to lowest priority, this approach requires waiting for the previous image to be transmitted before processing all images. While this approach can acquire the processed previous image quickly, it delays the processing of other images. Furthermore, this approach has a processing time of 20ms - 10ms + 13ms + 3ms × 5 = 38ms, which affects the real-time processing of images in the next cycle, necessitates increasing the number of image processing submodules, and reduces hardware resource utilization. In contrast, this approach has a processing time of 3ms × 5 + 13ms = 28ms, improving image processing efficiency without increasing the number of image processing submodules, thus improving hardware resource utilization.

[0119] Compared to the first-in-first-out (FIFO) scheduling method, which processes images sequentially according to their transmission order (taking 3ms × 5 + 13ms = 28ms to process the foreground images), this scheme takes only 3ms × 4 + 13ms = 25ms to process the foreground images. This reduces the image response time (e.g., AEB response time) by 3ms, thus lowering the processing latency of the foreground images.

[0120] In this embodiment, multiple sensors transmit their respective acquired images, with multiple images competing for hardware resources. This solution utilizes a first-in-first-out (FIFO) and priority-based parallel processing mechanism. Through hardware and software cooperation, it prioritizes the processing of high-priority tasks, ensuring processing efficiency and optimizing processing time. This allows the algorithm to acquire the processed front-view image at least 3ms earlier, reducing image response time and thus improving autonomous driving safety.

[0121] Based on the image processing method provided in the above embodiments, FIG6 is a schematic diagram of the structure of a chip provided in the present application. As shown in FIG6, the chip 70 (i.e., the chip 20 in FIG1 above) includes: a processor 701, a memory 702 and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, it implements the steps in the image processing method in the above embodiments.

[0122] Processor 701 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0123] In some embodiments, memory 702 can be an internal storage unit of chip 70, such as a hard disk or RAM of chip 70. In other embodiments, memory 702 can be an external storage device of chip 70, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on chip 70. Furthermore, memory 702 can include both internal storage units and external storage devices of chip 70. Memory 702 is used to store operating systems, applications, boot loaders, data, and other programs. Memory 702 can also be used to temporarily store data that has been output or will be output.

[0124] The chip and image processing method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0125] This application also provides a chip, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0126] This application also provides a controller, which includes the image processing device described in FIG1 or FIG2.

[0127] In this example, the controller can be a domain controller (DC), which is set in an integrated circuit. The integrated circuit may also include chips with other functions. The integrated circuit can be used in vehicles to ensure the normal operation of the vehicle.

[0128] This application also provides a vehicle, which includes the controller described above.

[0129] In this example, the vehicle could be a vehicle, a ship, or something similar that is equipped with multiple image sensors.

[0130] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0131] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0133] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0134] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: receiving multiple first storage messages; the first storage messages are used to instruct an image sensor to acquire images of a vehicle from different perspectives and complete transmission; when an image identifier carried in the first storage message indicates other images, the other images are sequentially read according to a preset reading strategy based on the storage location carried in the first storage message, and image processing is performed on the other images until a second storage message is received; wherein, the image identifier carried in the second storage message indicates the front view image with the highest priority; the front view image is read according to the storage location carried in the second storage message, and image processing is performed on the front view image; after the front view image is processed, the unprocessed other images are sequentially read according to the preset reading strategy, and the processing of the unprocessed other images is completed before receiving multiple first storage messages in the next cycle; wherein, the total processing time of the multiple other images and the front view image is less than or equal to the reciprocal of the frame rate of the image sensor, the total processing time reflects the image processing cycle, and the reciprocal of the frame rate reflects the image transmission cycle corresponding to the multiple first storage messages.

2. The method as described in claim 1, characterized in that, The preset reading strategy is either a first-in-first-out (FIFO) strategy or it is obtained by sorting according to the preset level corresponding to each image identifier.

3. The method as described in claim 1 or 2, characterized in that, The step of sequentially reading the other images according to the storage location carried by the first storage message and according to a preset reading strategy, and performing image processing on the other images until the second storage message is received, includes: sequentially reading N other images according to the storage locations carried by N first storage messages and according to the preset reading strategy, and simultaneously performing image processing on the N other images until the second storage message is received; wherein, N is a positive integer greater than 1, N represents the number of images processed simultaneously, and N is determined based on the image processing rate, the processing time of a single other image, the processing time of the front view image, and the frame rate of the image sensor, such that the sum of the processing times of multiple other images is greater than the difference between the transmission time of the front view image and the transmission time of a single other image, and the sum of the processing times of multiple other images and the front view image is less than or equal to the reciprocal of the frame rate.

4. The method as described in claim 3, characterized in that, The number of the second storage messages is M, where M is a positive integer greater than 1 and less than or equal to N; the step of reading the front view image according to the storage location of the front view image carried by the second storage message and performing image processing on the front view image includes: reading the M front view images sequentially according to the storage location of the front view image carried by each of the M second storage messages in a first-in-first-out strategy, and performing image processing on the M front view images simultaneously until the processing of the M front view images is completed.

5. The method as described in claim 3, characterized in that, The number of second storage messages is M, where M is a positive integer greater than N; the step of reading the front view image according to the storage location of the front view image carried by the second storage message and performing image processing on the front view image includes: according to the storage location of the front view image carried by each of the M second storage messages, reading N front view images sequentially according to a first-in-first-out strategy, and performing image processing on the N front view images simultaneously until the processing of the M front view images is completed.

6. The method as described in claim 4 or 5, characterized in that, The plurality of other images include at least one of a rear view image, a surround view image, a side view image, and an interior view image in a driving scene; the M front view images include a telephoto image and a wide-angle image in a forward-looking perspective.

7. The method as described in claim 1 or 2, characterized in that, The method further includes: receiving a second storage message; the image identifier carried in the second storage message indicates the highest priority front view image; reading the front view image according to the storage location of the front view image carried in the second storage message, and performing image processing on the front view image; after the processing of the front view image is completed, reading multiple other images sequentially according to the storage locations carried in multiple received first storage messages and following a preset reading strategy, and completing the processing of the multiple other images before receiving the second storage message of the next cycle; wherein, the image identifier carried in the first storage message indicates the other images.

8. A chip, characterized in that, The chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-7.

9. An image processing device, characterized in that, The image processing device includes a video input module and the chip as described in claim 8; the video input module is configured to receive other images transmitted by each of the plurality of first image sensors, and to receive a front view image transmitted by a second image sensor; Each of the other images is transmitted and a first storage message is generated, and a second storage message is generated after the front view image is transmitted; the plurality of first image sensors are used to acquire at least one of the rear view image, surround view image, side view image and interior view image in the driving scene, and the second image sensor is used to acquire the front view image in the driving scene; the chip is used to receive the plurality of first storage messages; According to the storage location carried in the first storage message, the other images are read sequentially according to a preset reading strategy, and image processing is performed on the other images until a second storage message is received; wherein, the image identifier carried in the second storage message indicates the front view image with the highest priority; according to the storage location of the front view image carried in the second storage message, the front view image is read, and image processing is performed on the front view image; after the processing of the front view image is completed, the other unprocessed images are read sequentially according to the preset reading strategy, and the processing of the other unprocessed images is completed before receiving multiple first storage messages in the next cycle; wherein, the sum of the processing time of the multiple other images and the front view image is less than or equal to the reciprocal of the frame rate of the image sensor, the sum of the processing time reflects the image processing cycle, and the reciprocal of the frame rate reflects the image transmission cycle corresponding to the multiple first storage messages.

10. The device according to claim 9, characterized in that, The video input module is further configured to store the other images transmitted by the plurality of first image sensors into a plurality of first preset image storage spaces after receiving the other images transmitted by the plurality of first image sensors respectively. Each of the first storage messages is generated based on the image identifiers used to indicate the image source and the storage locations used to indicate the first preset image storage space.

11. The device according to claim 9, characterized in that, The video input module is further configured to store the front view image in a second preset image storage space after receiving the front view image transmitted by the second image sensor; and generate the second storage message according to the image identifier used to indicate the image source and the storage location used to indicate the second preset image storage space.

12. The device according to claim 11, characterized in that, The number of the second image sensors is M; the video input module is also used to receive the front view images transmitted by each of the M second image sensors, and generate M second storage messages after the transmission of the M front view images is completed.

13. A controller, characterized in that, The controller includes the image processing device as described in any one of claims 9-12.

14. A vehicle, characterized in that, The vehicle includes the controller as described in claim 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images