Image optimization method, device and storage medium based on surround view images

Abstract

The application discloses a kind of based on the image optimization method, equipment and storage medium of ring view image, the image optimization method based on the ring view image includes: obtaining the target acquisition parameter of target ring view camera in each ring view camera;Target acquisition parameter is synchronized to each ring view camera;Each local image obtained by the image acquisition of each ring view camera with target acquisition parameter is acquired;Each local image is carried out image fusion processing, and the ring view image is obtained.The above scheme can optimize the image quality of ring view image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image optimization method and device based on a surround view image and a storage medium. BACKGROUND

[0002] A surround view image, also known as a panoramic image, is usually obtained by image stitching and fusion of images captured by multiple cameras (such as fisheye lenses). The application scenarios of surround view images are very diverse, such as panoramic cameras, panoramic drones, AVM (Around View Monitor) of smart cars, etc.

[0003] Taking AVM as an example, multiple surround view cameras are usually arranged around a car to fully capture the environment images of the car. However, due to the differences in installation positions and uneven effects of environmental light, the display effects of images captured by different surround view cameras are inconsistent, which affects the fusion effect of the surround view image.

[0004] Currently, existing methods perform color correction on each surround view camera to ensure that the images captured by each surround view camera are consistent. However, traditional color correction schemes rely on independent sensors or offline calibration methods, which are difficult to adapt to dynamic light changes in actual application scenarios (such as tunnel entry and exit, day and night switching, etc.). Therefore, there is an urgent need for an efficient and universal image optimization method to optimize the quality of surround view images. SUMMARY

[0005] The present application provides at least an image optimization method and device based on a surround view image, and a computer readable storage medium.

[0006] The first aspect of the present application provides an image optimization method based on a surround view image, comprising: obtaining a target acquisition parameter of a target surround view camera in each surround view camera; synchronizing the target acquisition parameter to each surround view camera; obtaining a local image obtained by image acquisition of each surround view camera with the target acquisition parameter; and performing image fusion processing on each local image to obtain the surround view image.

[0007] In an embodiment, the obtaining of the target acquisition parameter of the target surround view camera in each surround view camera comprises: obtaining state data of each surround view camera; performing election processing on each surround view camera according to the state data to obtain the target surround view camera; and determining the target acquisition parameter according to the image acquisition parameter of the target surround view camera.

[0008] In an embodiment, the state data comprises illumination data, signal-to-noise ratio data and color data, and the selecting each surround camera according to the state data to obtain the target surround camera comprises: performing weighted sum processing on the illumination data, the signal-to-noise ratio data and the color data of each surround camera respectively to obtain a priority of each surround camera; and determining the target surround camera from each surround camera according to the priority.

[0009] In an embodiment, after the selecting each surround camera according to the state data to obtain the target surround camera, the method further comprises: obtaining a parameter synchronization signal initiated by the target surround camera, the parameter synchronization signal comprising a to-be-synchronized address of a to-be-synchronized camera in each surround camera and an image acquisition parameter of the target surround camera; and transmitting the image acquisition parameter to the to-be-synchronized address.

[0010] In an embodiment, the determining the target acquisition parameter according to the image acquisition parameter of the target surround camera comprises: obtaining a personalized parameter of a user; and determining the target acquisition parameter according to the personalized parameter and the image acquisition parameter.

[0011] In an embodiment, after the synchronizing the target acquisition parameter to each surround camera, the method further comprises: detecting whether the target acquisition parameter is successfully synchronized to each surround camera; if yes, obtaining a local image collected by each surround camera according to the target acquisition parameter; and if no, re-synchronizing the acquisition parameter to each surround camera.

[0012] In an embodiment, the detecting whether the target acquisition parameter is successfully synchronized to each surround camera comprises: detecting whether each surround camera responds to the target acquisition parameter and / or detecting whether a communication process with each surround camera is abnormal; and in response to at least one surround camera not responding to the target acquisition parameter and / or the communication process with at least one surround camera being abnormal, determining that the target acquisition parameter is not successfully synchronized to each surround camera.

[0013] In an embodiment, the detecting whether each surround camera responds to the target acquisition parameter comprises: detecting whether an acknowledgement signal or a no-acknowledgement signal of the surround camera is received every time a preset number of target acquisition parameters are transmitted; in response to the acknowledgement signal being received, determining that the surround camera responds to the target acquisition parameter; and in response to the no-acknowledgement signal being received, determining that the surround camera does not respond to the target acquisition parameter.

[0014] The second aspect of the present application provides an image optimization device based on a surround view image, comprising: a parameter acquisition module, configured to acquire a target acquisition parameter of a target surround view camera in each surround view camera; a parameter synchronization module, configured to synchronize the target acquisition parameter to each surround view camera; an image acquisition module, configured to acquire a local image obtained by image acquisition of each surround view camera with the target acquisition parameter; and an image fusion module, configured to perform image fusion processing on each local image to obtain the surround view image.

[0015] The third aspect of the present application provides an electronic device, comprising a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned image optimization method based on a surround view image.

[0016] The fourth aspect of the present application provides a computer-readable storage medium, having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the above-mentioned image optimization method based on a surround view image.

[0017] The above-mentioned scheme, by acquiring a target acquisition parameter of a target surround view camera in each surround view camera, synchronizing the target acquisition parameter to each surround view camera, allowing surround view cameras at different spatial positions to be configured with a unified target acquisition parameter, eliminating color deviation caused by traditional time-sharing processing. Acquiring a local image obtained by image acquisition of each surround view camera with the target acquisition parameter, completing color parameter injection and achieving parameter unification in the image sensor data output stage, obtaining each local image with consistent color. Image processing based on the Raw domain avoids post-processing techniques used in traditional methods, which can reduce the load of controller operation resources and improve image quality. Thus, by performing image fusion processing on each local image, a surround view image with consistent color and brightness can be obtained.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of the image optimization method based on a surround view image of the present application;

[0021] Figure 2 is an exemplary surround view system architecture diagram in the image optimization method based on a surround view image of the present application;

[0022] Figure 3 is a block diagram of the image optimization device based on a surround view image shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a structural schematic diagram of an embodiment of an electronic device of the present application.

[0024] Figure 5 is a structural schematic diagram of an embodiment of a computer readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0026] In the following description, specific details such as specific system structures, interfaces, techniques, etc. are presented in order to provide a thorough understanding of the present application for the sake of explanation, but not for the sake of limitation.

[0027] The term "and / or" herein is merely a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally represents that the front and rear associated objects are in an "or" relationship. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] It should be noted that the image optimization method based on surround view images of the present application can be used in various application scenarios involving surround view cameras, such as panoramic cameras, panoramic drones, AVMs of smart cars, etc., which are not limited here.

[0029] The application scenarios of the embodiments of the present application are mainly taken as examples of the application of AVMs of cars. For the sake of understanding, one of the application scenarios of the present application will be described by way of example.

[0030] The existing vehicle-mounted surround view system usually has the following problems: due to the difference in installation position, the surround view cameras are not uniformly affected by the environmental light, resulting in inconsistent color temperature and brightness of the spliced images, which affects the fusion effect of the panoramic picture. The traditional color correction scheme relies on independent sensors or offline calibration, which is difficult to adapt to dynamic light changes (such as vehicle entry and exit in a tunnel, day and night switching, etc.). The color and brightness consistency method based on post-processing has problems such as performance and color distortion.

[0031] Please refer to Figure 1 , Figure 1 is a flowchart of an exemplary embodiment of the image optimization method based on surround view images of the present application. Specifically, it can include the following steps:

[0032] Step S110, obtaining target acquisition parameters of a target surround camera in each surround camera.

[0033] It can be understood that in order to obtain a surround image (panoramic image), a plurality of surround cameras are usually arranged to respectively acquire images, and the images acquired by each surround camera are spliced and fused. In a common method, the surround camera is usually a fisheye lens (fisheye camera), and each surround camera is independently arranged and acquires images, so it is difficult to ensure the consistency of the images acquired by each surround camera, that is, the image quality of the surround image cannot be ensured.

[0034] Exemplarily, in the application scenario of the present application, a surround system can be arranged as shown in FIG. Figure 2 Figure 2 is an exemplary surround system architecture in the image optimization method based on a surround image of the present application. In the surround system architecture, a bus type I2C bus architecture is used to construct a multi-camera master-slave communication network, and a master controller (MCU) is connected to each surround camera node through a standard I2C interface (SDA / SCL). Each surround camera node includes a master camera node (master node) and a slave camera node (slave node). The master node is responsible for initiating communication, and the slave node responds to the request. Each camera is configured with an independent I2C address code, for example, the master node address is set to 0x40, and the slave nodes are 0x41, 0x42, and 0x43 in turn, to avoid address conflicts. A ring topology structure is established through hardware pins (such as GPIO ground or power supply) or software registers, and low communication is ensured. The hardware layer can integrate an ISP chip with EEPROM to realize non-volatile storage of gain parameters.

[0035] In the surround system architecture, the master camera node corresponds to the target surround camera. The method of the present application can be applied to the master controller, or to other controllers (such as domain controllers, etc.) that have a communication connection with the master controller, which is not limited here. In order to realize the communication of the above architecture, the present application defines at least three kinds of data frame structures and uses a special communication protocol stack for the image optimization method proposed and the above architecture. For example, a parameter request frame (0xA1) is used to initiate a parameter synchronization instruction. A gain data frame (0xB2) includes red and blue channel gain ratio (12-bit precision), white balance matrix data, etc. A state confirmation frame (0xC3) includes CRC check code and device response state, etc.

[0036] Through the above architecture, the target surround camera (master camera node) in each surround camera can be determined, and the target acquisition parameters of the target surround camera can be obtained. The target acquisition parameters refer to the acquisition parameters used by the target surround camera to acquire images, which can include but are not limited to white balance Rgain and Bgain, exposure parameters, etc., which are not limited here.

[0037] ​S120, synchronizing the target acquisition parameter to each surround camera.

[0038] In combination with the foregoing steps, after obtaining the target acquisition parameter, it can be synchronized to each surround camera. Among them, it can be to synchronize the parameters to all surround cameras, or to synchronize the parameters to other surround cameras (slave nodes) except the target surround camera in each surround camera, or to select part of the surround cameras from all surround cameras for parameter synchronization, etc. Herein, no limitation is made.

[0039] Exemplarily, all devices in the current communication network can be quickly notified by the way of broadcast communication. Among them, the communication address of each surround camera can be obtained in advance or in real time, and herein no limitation is made. For example, the master node can send a start signal for parameter synchronization, broadcast the slave node address (including a write flag) requiring parameter synchronization, and transmit the target acquisition parameter such as Rgain and Bgain gain ratio and exposure parameter. The master node can achieve single transmission of the master node and synchronous reception of multiple slave nodes by sending a global broadcast address, reducing the bus occupation time.

[0040] Another exemplarily, in this step, the target acquisition parameter can be synchronized to each surround camera, or after a certain parameter adjustment, it is synchronized to each surround camera, and herein no limitation is made. The specific parameter adjustment method can include but is not limited to parameter enhancement, parameter weakening, etc., which can be flexibly adjusted according to actual needs, and herein no further description is made.

[0041] S130, obtaining the local image obtained by each surround camera with the target acquisition parameter.

[0042] In combination with the foregoing steps, the target acquisition parameter (such as color temperature K value, white balance gain matrix, camera light metering mode, etc.) of the master node is synchronized to each slave node, and the slave node will forcibly overwrite the target acquisition parameter of its own ISP after receiving the target acquisition parameter, thereby realizing the parameter unification of each surround camera in the same AVM in the Raw domain, and obtaining the local image collected by each surround camera with the same target acquisition parameter. Image processing (such as exposure delay compensation algorithm, etc.) is realized at the firmware layer to eliminate the timing error caused by data transmission.

[0043] It should be noted that the local image refers to the image used to compose the local part of the surround image, which can actually be the original image collected by the surround camera, or the image after a certain image processing on the original image collected by the surround camera, and herein no limitation is made. The image processing method can be set as one or more as needed, and herein no further description is made.

[0044] Step S140: Perform image fusion processing on each local image to obtain a panoramic image.

[0045] Based on the preceding steps, after obtaining the images acquired by each surround-view camera with the same target acquisition parameters, these images can be fused and stitched together using image fusion methods specific to this technical field; details will not be elaborated here. Thus, a surround-view image with consistent color and brightness can ultimately be obtained and output.

[0046] As can be seen, this application obtains the target acquisition parameters of the target surround-view camera from each surround-view camera and synchronizes these parameters to each camera, enabling surround-view cameras at different spatial locations to be configured with unified target acquisition parameters, thus eliminating color deviations caused by traditional time-division processing. Local images obtained from each surround-view camera using the target acquisition parameters are acquired, and color parameters are injected during the image sensor data output stage to achieve parameter unification, resulting in consistent local images. Image processing based on the Raw domain avoids the post-processing techniques used in traditional methods, reducing the computational load on the controller and improving image quality. Therefore, image fusion processing of each local image yields a surround-view image without color or brightness banding.

[0047] Based on the above embodiments, this application embodiment describes the steps for obtaining target acquisition parameters of the target surround-view camera in each surround-view camera. Specifically, the method of this embodiment includes the following steps:

[0048] Acquire the status data of each surround-view camera; perform election processing on each surround-view camera based on the status data to obtain the target surround-view camera; determine the target acquisition parameters based on the image acquisition parameters of the target surround-view camera.

[0049] Referring to the foregoing embodiments, this application typically includes multiple surround-view cameras. The target surround-view camera is equivalent to the master node among the multiple surround-view cameras, and the other surround-view cameras are equivalent to the slave nodes among the multiple surround-view cameras. The master-slave relationship between the surround-view cameras can be preset or determined in real time, and is not limited here.

[0050] Here, "state data" refers to the state of the surround-view camera during image acquisition. State data may include, but is not limited to, one or more of the following: signal-to-noise ratio, illumination intensity, etc., without limitation here.

[0051] Exemplarily, the application can acquire state data of each surround-view camera to determine a target surround-view camera (master node) from each surround-view camera. The method of determining the target surround-view camera can refer to, for example, an election algorithm. Then, the image acquisition parameters of the currently determined target surround-view camera are taken as target acquisition parameters, or the target acquisition parameters are obtained by adjusting the image acquisition parameters of the target surround-view camera to a certain extent. The specific parameter adjustment process can be set on demand, which is not described here. Each camera node (including the master node and the slave node) can periodically actively broadcast its own state data through I2C, or the master node periodically broadcasts a parameter synchronization instruction, and the slave node responds to the parameter synchronization instruction and uploads its own state data.

[0052] Exemplarily, the process of determining the target surround-view camera in the application can be one-time, periodic, or responsive to a preset trigger condition, which is not limited here. For example, the election of the master node can be periodic, that is, the election process of each surround-view camera is performed according to the state data of each surround-view camera every preset period to obtain the target surround-view camera of the current period. The target surround-view cameras of different periods can be the same or different, which is not limited here. For another example, the election of the master node can be responsive to a preset trigger condition, for example, when the signal-to-noise ratio of the master node is greater than a preset signal-to-noise ratio threshold (such as 40db), a re-election is automatically triggered.

[0053] On the basis of the above embodiments, the application embodiment describes the step of performing an election process on each surround-view camera according to the state data to obtain a target surround-view camera. The state data includes illumination data, signal-to-noise ratio data, and color data. Specifically, the method of the embodiment includes the following steps:

[0054] The illumination data, signal-to-noise ratio data, and color data of each surround-view camera are respectively weighted and summed to obtain the priority of each surround-view camera; and a target surround-view camera is determined from each surround-view camera according to the priority.

[0055] In combination with the foregoing embodiments, the state data in the application can include illumination data (such as illumination intensity), signal-to-noise ratio data (such as signal-to-noise ratio), and color data (such as color histogram).

[0056] Exemplarily, the illumination uniformity can be determined according to the illumination intensity, and the color entropy can be determined according to the color histogram, which can be referred to the related methods in the art. Then, the illumination uniformity, signal-to-noise ratio, and color entropy of each surround-view camera can be respectively weighted and summed by using a pre-set weighting scoring mechanism to obtain the priority of each surround-view camera. Each state data can correspond to the same or different preset weight, which is not limited here.

[0057] For example, the mathematical expression of the weighted scoring mechanism can be: Priority = 0.4 × Illumination Uniformity + 0.3 × Signal-to-Noise Ratio + 0.3 × Color Entropy.

[0058] After obtaining the priorities of each surround-view camera, the surround-view camera with the highest priority can be selected as the target surround-view camera. Selecting the camera node with the best state as the master node ensures that the system continuously selects the best image signal data.

[0059] Another example is that the mathematical expression for the signal-to-noise ratio (SNR) can be:

[0060]

[0061] in, It is the variance of the image signal, the maximum value of the local neighborhood variance of the image pixels. This is the noise variance, the minimum value of the local neighborhood variance of image pixels. For specific principles, please refer to relevant technologies in this field, which will not be elaborated here. The statistical region for the signal-to-noise ratio (SNR) value can be calculated based on the deployment locations of each surround-view camera, selecting the ROI of overlapping regions in images acquired by adjacent surround-view cameras (adjacent images). The specific calculation method will not be elaborated here.

[0062] It should be noted that if a master node switch occurs, the bus reset signal (e.g., 9 consecutive SCL pulses) sent by the newly elected master node during the switch can be received, ensuring that the bus state is cleared.

[0063] Based on the above embodiments, this application embodiment describes the steps after selecting each surround-view camera according to state data to obtain the target surround-view camera. Specifically, the method of this embodiment includes the following steps:

[0064] Acquire the parameter synchronization signal initiated by the target surround view camera. The parameter synchronization signal includes the synchronization address of the camera to be synchronized in each surround view camera and the image acquisition parameters of the target surround view camera; transmit the image acquisition parameters to the synchronization address.

[0065] In conjunction with the foregoing embodiments, after determining the target surround-view camera among the surround-view cameras, the parameter synchronization signal (e.g., start signal) periodically or conditionally triggered by the target surround-view camera can be received. Then, the address to be synchronized of the target slave node that needs to be synchronized can be broadcast, and the image acquisition parameters of the target surround-view camera can be transmitted.

[0066] After receiving these signal data, the image acquisition parameters can be transmitted to the address to be synchronized, that is, the target acquisition parameters can be synchronized to the target slave node.

[0067] On the basis of the above embodiments, the embodiments of the present application explain the step of determining the target acquisition parameter according to the image acquisition parameter of the target surround view camera. Specifically, the method of the present embodiment comprises the following steps:

[0068] Obtaining the personalized parameter of the user; determining the target acquisition parameter according to the personalized parameter and the image acquisition parameter.

[0069] In combination with the foregoing embodiments, after determining the image acquisition parameter of the target surround view camera, the image acquisition parameter can be directly determined as the target acquisition parameter, or the image acquisition parameter can be adjusted to obtain the target acquisition parameter.

[0070] Illustratively, the personalized parameter of the user can be obtained. The personalized parameter refers to the image acquisition parameter or image color parameter customized by the user, which represents the user's preference for the color style of image display. Therefore, the target acquisition parameter can be determined according to the personalized parameter and the image acquisition parameter.

[0071] For example, the personalized parameter of the user is a bright style (the specific parameter details can be adjusted as needed, such as increasing exposure, etc., which is not limited here), and the exposure value can be increased based on the image acquisition parameter. Or it is judged whether the image is abnormal (such as overexposure) after adjusting the image acquisition parameter according to the personalized parameter; if not, the target acquisition parameter is determined according to the personalized parameter and the image acquisition parameter; if so, the target acquisition parameter is determined according to the image acquisition parameter.

[0072] Another illustratively, the above example process can be applied to adjust the image acquisition parameter of the surround view camera according to the personalized parameter of the user to obtain the target acquisition parameter; it can also be selected to be applied to adjust the image parameter (such as color, brightness, etc.) of the surround view image after obtaining the surround view image according to the personalized parameter of the user to obtain the target surround view image for final display. The related image processing method can refer to the image processing technology in the art, which is not described here.

[0073] Specifically, the present application can design a human-computer interaction system based on, for example, Qt framework, in actual application scenarios, to provide an interaction interface for AVM users (such as drivers), allowing the driver to adjust the camera acquisition parameter and / or the color of the entire image in real time according to the actual scene or subjective feeling. For example, it can include but is not limited to: a real-time color temperature adjustment module (for example, the color temperature range can be 2800K-6500K); a color contrast dynamic curve (for example, 8 adjustable segments); a multi-view synchronous preview window (for example, supporting synchronous display of images of 4 surround view cameras).

[0074] The interaction system can also be designed in three operation modes, for example: basic mode: 5 preset scene modes are provided for user selection (such as tunnel / overcast / sunny light, etc.); professional mode: open RGB independent gain adjustment (such as ±15% range); engineering mode: with parameter export and OTA upgrade function, etc.

[0075] Through the above example of the interaction interface, the driver can configure the color parameters defined in the interaction interface, and combine the parameters of multiple slave node cameras to perform personalized brightness increase / decrease and color balance optimization, thereby improving the color quality of the image.

[0076] Further, a trained preference analysis model (which can be obtained based on LSTM (Long Short-Term Memory Network) training) can also be pre-set in the interaction system. When the user sets the personalized parameters each time, the preference features of the user adjusting the color parameters of the interaction interface can be extracted, and the image features (such as image color features, image content features, etc.) collected when adjusting the parameters can also be extracted, and the multi-modal time series data is processed and analyzed. The specific implementation process can include but is not limited to: data preprocessing (standardization and dimensionality reduction), LSTM unit construction (input gate, forget gate, output gate), and feature fusion (weight learning mechanism), and finally output the color preference vector of the user. Subsequently, the user behavior features and / or image features obtained can be directly input into the preference analysis model to obtain the parameter configuration recommended by the preference analysis model (which can include image acquisition parameters and / or image display parameters, etc.). The return of the preference color parameters can use an asymmetric encryption algorithm (such as RSA algorithm), and the signature mechanism is realized by a hash function and a digital certificate to write a verification before writing, so as to ensure the integrity and tamper resistance of the data.

[0077] On the basis of the above embodiment, the embodiment of the present application describes the steps after synchronizing the target acquisition parameters to each surround camera. Specifically, the method of the present embodiment includes the following steps:

[0078] Detecting whether the target acquisition parameters are successfully synchronized to each surround camera; if yes, acquiring the local images collected by each surround camera according to the target acquisition parameters; if not, re-synchronizing the acquisition parameters to each surround camera.

[0079] In combination with the foregoing embodiment, the method of the present application can also set a fault detection mechanism for the parameter synchronization process to ensure that the acquisition parameters can be successfully written into each surround camera.

[0080] Exemplarily, the fault detection mechanism can be performed after each I2C communication. For example, it can be detected whether there is an exception in the I2C communication process, and / or whether there is an exception in receiving the target acquisition parameter by the slave node. If the parameter synchronization is successful, the local images acquired by each surround view camera according to the target acquisition parameter can be obtained according to the foregoing examples. If the parameter synchronization fails, the acquisition parameter can be re-synchronized to each surround view camera, and / or the fault information of this time can be recorded for subsequent analysis.

[0081] On the basis of the foregoing embodiment, the step of detecting whether the target acquisition parameter is successfully synchronized to each surround view camera is described in the embodiment of the present application. Specifically, the method of the embodiment comprises the following steps:

[0082] It is detected whether each surround view camera responds to the target acquisition parameter, and / or whether there is an exception in the communication process with each surround view camera. In response to at least one surround view camera not responding to the target acquisition parameter, and / or there being an exception in the communication process with at least one surround view camera, it is determined that the target acquisition parameter is not successfully synchronized to each surround view camera.

[0083] In combination with the foregoing embodiment, it is explained that, under normal circumstances, the slave node will return an ACK signal (acknowledgement signal) after receiving the target acquisition parameter, to indicate that it responds to the parameter synchronization process. And there should be a change between high and low levels in the communication process. If at least one surround view camera does not respond to the target acquisition parameter, and / or there is an exception in the communication process with at least one surround view camera, it can be determined that the target acquisition parameter is not successfully synchronized to each surround view camera.

[0084] For example, the method of detecting whether the target acquisition parameter is successfully synchronized to each surround view camera (the fault detection mechanism) can be to read the status register (such as I2C_SR) of the I2C controller to detect the following key flags: 1, NACK (no response): the slave node does not respond to the address or data frame (judged by the I2C_ERROR_NACK flag). 2, BUS ERROR: illegal start / stop condition (such as arbitration loss). 3, TIMEOUT: the SCL line continuously low for more than a preset timeout threshold (such as 30 clock cycles).

[0085] On the basis of the foregoing embodiment, the step of detecting whether each surround view camera responds to the target acquisition parameter is described in the embodiment of the present application. Specifically, the method of the embodiment comprises the following steps:

[0086] Each transmission of a preset number of target acquisition parameters, detect whether a response signal or no response signal of the surround view camera is received; in response to receiving the response signal, determine that the surround view camera responds to the target acquisition parameter; in response to receiving the no response signal, determine that the surround view camera does not respond to the target acquisition parameter.

[0087] In combination with the foregoing embodiments, the fault detection mechanism of the present application can be based on the response mechanism setting of the slave node. For example, in the parameter synchronization process, after each byte of data (which can be the target acquisition parameter) is transmitted, it can be detected whether the slave node returns an ACK signal. If the slave node fails to check the received data (for example, CRC check), a NACK signal returned by the slave node can be received. After receiving the NACK signal, the master node can be triggered to retransmit the target acquisition parameter.

[0088] It should be further pointed out that the fault detection mechanism of the present application can also be to detect the images collected by each surround view camera with the synchronized image acquisition parameters (target acquisition parameters) after the parameter synchronization is completed, and analyze the image quality of these images in the same way. Taking the signal-to-noise ratio that can be used to measure the image quality as an example, if the signal-to-noise ratio is greater than a signal-to-noise ratio threshold, the backup parameters (which can be preset and / or stored one or more sets of historical image acquisition parameters) can be automatically switched to, so that each surround view camera collects images with the backup parameters.

[0089] As can be seen from the above example method, the present application constructs a distributed parameter synchronization architecture through the I2C bus, adopts a mechanism of adaptively determining a master node and dynamically transmitting the gain ratio of the red channel and the blue channel, and implements linear transformation processing in the RAW domain. The surround view cameras at different spatial positions are allowed to share and apply the same set of image acquisition parameters, and the color deviation caused by traditional time-sharing processing is eliminated, which can improve the color consistency after image fusion and improve the image quality.

[0090] The front-end preprocessing unit integrates the parameter processing module at the camera end, and the color parameter injection is completed in the image sensor data output stage to realize parameter unification, avoid the use of post-processing technology, reduce the control of the controller operation resource load, and improve the real-time performance.

[0091] Through the multi-dimensional weighted scoring mechanism and the fault detection mechanism, the optimal image signal data is selected as the optimal frame reference data for color consistency processing. At the same time, the fault detection module can adopt an abnormal data isolation mechanism to automatically switch to the backup data stream when it is detected that the sensor noise value exceeds the threshold. The mechanism can realize mode switching through a finite state machine, and ensure that the color difference is still small in the scene with large illumination color difference.

[0092] In addition, the application introduces a brightness and color space mapping engine to realize quantitative conversion of driver subjective preference parameters. User adjustment records under different lighting conditions are recorded through an adaptive learning algorithm, a personalized color feature library is established, and real-time parameter rewriting and previewing are supported based on an OpenGL ES pipeline rendering.

[0093] It should be further explained that the execution subject of the image optimization method based on the surround view image can be an image optimization device based on the surround view image. For example, the image optimization method based on the surround view image can be executed by a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the image optimization method based on the surround view image can be realized by a processor calling computer readable instructions stored in a memory.

[0094] Figure 3 is a block diagram of an image optimization device based on a surround view image according to an example embodiment of the present application. As shown in Figure 3 The example image optimization device based on the surround view image 300 includes a parameter acquisition module 310, a parameter synchronization module 320, an image acquisition module 330, and an image fusion module 340. Specifically:

[0095] The parameter acquisition module 310 is configured to acquire a target acquisition parameter of a target surround view camera in each surround view camera.

[0096] The parameter synchronization module 320 is configured to synchronize the target acquisition parameter to each surround view camera.

[0097] The image acquisition module 330 is configured to acquire a local image obtained by image acquisition of each surround view camera at the target acquisition parameter.

[0098] The image fusion module 340 is configured to perform image fusion processing on each local image to obtain a surround view image.

[0099] In the exemplary image optimization apparatus based on the surround view image, the target acquisition parameter of the target surround camera is obtained, and the target acquisition parameter is synchronized to each surround camera, so that the surround cameras at different spatial positions can be configured with the unified target acquisition parameter, and the color deviation caused by the traditional time-sharing processing is eliminated. The local images obtained by each surround camera using the target acquisition parameter for image acquisition are obtained, and the color parameter injection is completed in the image sensor data output stage to realize parameter unification, and the local images with consistent colors are obtained. The image processing based on the Raw domain avoids the post-processing technology used in the traditional method, can reduce the operation resource load of the controller, and improves the image quality. Therefore, the image fusion processing is performed on each local image, and the surround view image with consistent color and brightness is obtained.

[0100] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, and will not be described here. The apparatus provided in the above embodiments can be divided into different functional modules according to the needs in actual application, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0101] The functions of each module can be referred to the embodiments of the image optimization method based on the surround view image, and will not be described here.

[0102] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of an electronic device. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is configured to execute program instructions stored in the memory 101 to implement the steps in any of the above-described image optimization method embodiments based on the surround view image. In one specific implementation scenario, the electronic device 100 can include but is not limited to a microcomputer, a server, and in addition, the electronic device 100 can also include a notebook computer, a tablet computer, and other mobile devices, which are not limited here.

[0103] Specifically, the processor 102 is configured to control itself and the memory 101 to implement the steps in any of the above-mentioned embodiments of the image optimization method based on the surround view image. The processor 102 can also be referred to as a CPU (Central Processing Unit). The processor 102 can be an integrated circuit chip with processing capability. The processor 102 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 102 can be implemented by an integrated circuit chip jointly.

[0104] In the example electronic device, the target acquisition parameter of the target surround view camera is obtained, and the target acquisition parameter is synchronized to each surround view camera, so that the surround view cameras at different spatial positions can be configured with the unified target acquisition parameter, and the color deviation caused by the traditional time-division processing is eliminated. The local images obtained by each surround view camera using the target acquisition parameter for image acquisition are obtained, and the color parameter injection is completed in the image sensor data output stage to achieve parameter unification, and the local images with consistent colors are obtained. The image processing based on the Raw domain avoids the post-processing technology used in the traditional method, can reduce the operation resource load of the controller, and improves the image quality. Thus, the image fusion processing is performed on each local image, and the surround view image with consistent color and brightness is obtained.

[0105] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 110 stores program instructions 111 capable of being executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned embodiments of the image optimization method based on the surround view image.

[0106] In the exemplary storage medium, by running the program instructions in the storage medium, the target acquisition parameter of the target surround camera among the surround cameras is acquired, and the target acquisition parameter is synchronized to the surround cameras, so that the surround cameras at different spatial positions can be configured with uniform target acquisition parameters, and color deviation caused by traditional time-sharing processing is eliminated. The local images obtained by the surround cameras at the target acquisition parameter are acquired, and the color parameter injection is completed to realize parameter unification in the image sensor data output stage, and the local images with consistent colors are obtained. The image processing based on the Raw domain avoids the post-processing technology used in the traditional method, can reduce the operation resource load of the controller, and improves the image quality. Therefore, the image fusion processing is performed on the local images, and the surround image with consistent color and brightness without discontinuity is obtained.

[0107] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0108] The above description of each embodiment tends to emphasize the differences between each embodiment, and the same or similar parts can be mutually referred to, and for brevity, will not be repeated here.

[0109] In several embodiments provided in the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the above-described device implementation is only schematic, for example, the division of the module or unit is only a logical function division, and the actual implementation can have another division manner, for example, the unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0110] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit. When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the method of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. An image optimization method based on surround view images, characterized by, The method comprises: acquiring target acquisition parameters of a target surround-view camera among the surround-view cameras; The step of acquiring the target acquisition parameters of the target surround-view camera among the surround-view cameras comprises: acquiring state data of the surround-view cameras in the image acquisition process, the state data comprising illumination data, signal-to-noise ratio data and color data; performing election processing on the surround-view cameras according to the state data to obtain the target surround-view camera; and determining the target acquisition parameters according to the image acquisition parameters of the target surround-view camera; The step of performing election processing on the surround-view cameras according to the state data to obtain the target surround-view camera comprises: performing weighted summation processing on the illumination data, the signal-to-noise ratio data and the color data of each surround-view camera respectively to obtain the priority of each surround-view camera; and determining the target surround-view camera from the surround-view cameras according to the priority; The step of performing weighted summation processing on the illumination data, the signal-to-noise ratio data and the color data of each surround-view camera respectively to obtain the priority of each surround-view camera comprises: determining the illumination uniformity according to the illumination data; determining the color entropy according to the color data; and performing weighted summation processing on the illumination uniformity, the signal-to-noise ratio data and the color entropy of each surround-view camera respectively to obtain the priority of each surround-view camera; The step of determining the target surround-view camera from the surround-view cameras according to the priority comprises: determining the surround-view camera with the highest priority from the surround-view cameras as the target surround-view camera; synchronizing the target acquisition parameters to the surround-view cameras; acquiring local images obtained by the surround-view cameras performing image acquisition according to the target acquisition parameters; performing image stitching and fusion processing on the local images to obtain the surround view image.

2. The method of claim 1, wherein, After the step of performing election processing on the surround-view cameras according to the state data to obtain the target surround-view camera, the method further comprises: acquiring a parameter synchronization signal initiated by the target surround-view camera, the parameter synchronization signal comprising a to-be-synchronized address of a to-be-synchronized camera among the surround-view cameras and image acquisition parameters of the target surround-view camera; transmitting the image acquisition parameters to the to-be-synchronized address.

3. The method of claim 1, wherein, The step of determining the target acquisition parameters according to the image acquisition parameters of the target surround-view camera comprises: acquiring personalized parameters of a user; determining the target acquisition parameters according to the personalized parameters and the image acquisition parameters.

4. The method of claim 1, wherein, After the step of synchronizing the target acquisition parameters to the surround-view cameras, the method further comprises: detecting whether the target acquisition parameters are successfully synchronized to the surround-view cameras; if yes, acquiring local images acquired by the surround-view cameras according to the target acquisition parameters; if no, re-synchronizing the acquisition parameters to the surround-view cameras.

5. The method of claim 4, wherein, The step of detecting whether the target acquisition parameters are successfully synchronized to the surround-view cameras comprises: detecting whether the surround-view cameras send responses to the target acquisition parameters and / or detecting whether there is an abnormality in the communication process with the surround-view cameras; determining that the target collection parameter is not successfully synchronized to each surround camera in response to at least one surround camera not responding to the target collection parameter and / or an abnormality in a communication process with at least one surround camera.

6. The method of claim 5, wherein, The detecting whether each surround camera responds to the target collection parameter comprises: detecting whether a response signal or a no-response signal of the surround camera is received per a preset number of target collection parameters transmitted; determining that the surround camera responds to the target collection parameter in response to receiving the response signal; determining that the surround camera does not respond to the target collection parameter in response to receiving the no-response signal.

7. An electronic device, comprising: a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method of any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon program instructions, wherein, The program instructions, when executed by the processor, implement the method of any one of claims 1 to 6.

Images