An image processing method and apparatus

Abstract

The application discloses an image processing method and device, which realizes RAW image-oriented encoding compression, is suitable for various existing encoding standards, and can guarantee the image quality after compression. In the method, a terminal device can acquire a first original RAW image; the first RAW image is preprocessed to obtain a second RAW image, the second RAW image comprises an image in YCoCgDg format, Y is a luminance component, Co and Cg are color components, and Dg is a difference component; the second RAW image is encoded to obtain a first encoding code stream; and the first encoding code stream is stored or transmitted to a cloud device.

Description

Image processing method and device TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and device. BACKGROUND

[0002] Traditional image signal processing (ISP) devices are often tightly coupled with image acquisition devices (for example, camera sensors). After a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS) in the image acquisition device capture image irradiance records in a Bayer mode as a raw image format (also referred to as a raw image or RAW image), the ISP device converts the RAW image into an RGB image through a series of linear and nonlinear steps, and stores or transmits the compressed RGB image to a device with a function of processing visual tasks.

[0003] At present, the device with the function of processing visual tasks mainly implements processing of the RGB image through various machine vision models. On the one hand, if the corresponding machine vision model is trained and tested using the RGB image generated by different ISP devices, the image domain difference is inevitable, thereby adversely affecting the inference accuracy. At the same time, some original information is lost after the RAW image is converted into the RGB image, which also affects the inference accuracy of the machine vision model. On the other hand, the traditional ISP device is designed for human eye vision rather than machine vision, and the sensitive frequency domain range has limitations. On the other hand, in some scenarios, if the RGB image needs to be compressed and decompressed at a large magnification, the image quality will also be greatly reduced.

[0004] Therefore, there is an urgent need for an encoding compression method for RAW images, which is suitable for various existing encoding standards and can guarantee the image quality after compression.

[0005] SUMMARY

[0006] The present application provides an image processing method and device for realizing encoding compression for RAW images, which is suitable for various existing encoding standards and can guarantee the image quality after compression.

[0007] In a first aspect, the present application provides an image processing method, which can be executed by a terminal device, such as a smartphone, a vehicle, or other types of terminal devices, which are not limited in the embodiments of the present application. The method can include: obtaining a first original RAW image; pre-processing the first RAW image to obtain a second RAW image, the second RAW image including an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; encoding the second RAW image to obtain a first encoded bitstream; storing the first encoded bitstream, or sending the first encoded bitstream to a cloud device.

[0008] Through the above method, the terminal device can pre-process the original RAW image, and the obtained RAW image can be adapted to various existing encoding standards, so as to multiplex the existing encoding standards to encode and compress the image, thereby guaranteeing the image compression quality. At the same time, the limitation of the human eye sensitive frequency domain range can be overcome, thereby adapting to various machine vision models and improving the inference accuracy of related models.

[0009] In a possible implementation, the pre-processing of the first RAW image to obtain a second RAW image includes: performing color domain transformation on the first RAW image to obtain images of four components in YCoCgDg format; and splicing the images of the four components according to a set splicing mode to obtain the second RAW image. It should be understood that the YCoCgDg format is only an example and is not limited in any way. In other embodiments, the second RAW image can also use other color domain formats and be pre-processed by using a method similar to the present application.

[0010] In a possible implementation, the set splicing mode includes any one of the following: a matrix splicing mode, a horizontal splicing mode, and a vertical splicing mode; and the shape of the second RAW image includes any one of the following: a grid, a horizontal strip, and a vertical strip.

[0011] In a possible implementation, the color domain transformation on the first RAW image to obtain images of four components in YCoCgDg format includes: performing nonlinear transformation on the first RAW image to obtain a transformed first RAW image; and performing color domain transformation on the transformed first RAW image to obtain images of four components in YCoCgDg format.

[0012] In a possible implementation, the pre-processing of the first RAW image to obtain a second RAW image further includes: performing desensitization detection and desensitization processing based on an image of a luminance component in the four components to obtain a desensitized image of the luminance component; and the splicing, according to a set splicing manner, of the images of the four components to obtain the second RAW image includes: splicing, according to the set splicing manner, the desensitized image of the luminance component, images of color components in the four components, and an image of a difference component to obtain the second RAW image.

[0013] In a possible implementation, the method further includes: obtaining the first encoded code stream from a storage medium; decoding a second RAW image from the first encoded code stream; performing desensitization detection and desensitization processing based on the second RAW image to obtain a third RAW image; encoding the third RAW image to obtain a second encoded code stream; and sending the second encoded code stream to the cloud device.

[0014] In a possible implementation, the performing desensitization detection and desensitization processing based on the second RAW image to obtain a third RAW image includes: performing desensitization detection and desensitization processing based on an image of a luminance component in the second RAW image to obtain a desensitized image of the luminance component; and splicing, according to a set splicing manner, the desensitized image of the luminance component, images of color components in the second RAW image, and an image of a difference component to obtain the third RAW image.

[0015] In a possible implementation, the encoding of the second RAW image to obtain a first encoded code stream includes: encoding the second RAW image to obtain the first encoded code stream by using any one of the following encoding standards: an H264 encoding standard, an H265 encoding standard, an H266 encoding standard, and a JPEG low-latency encoding standard.

[0016] In a possible implementation, the terminal device includes an image acquisition module, and the obtaining of the first RAW image includes: acquiring, by the image acquisition module, a RAW image corresponding to a current target scene to obtain the first RAW image; or obtaining the first RAW image from a storage medium.

[0017] In a possible implementation, the terminal device further includes an image signal processing (ISP) module, and the method further includes: processing, by the ISP module, the first RAW image to obtain a fourth image, the fourth image including an image in YUV format, Y being a luminance component, and U and V being color components; encoding the fourth image to obtain a third encoded code stream; and storing the third encoded code stream or sending the third encoded code stream to the cloud device.

[0018] In a possible implementation, the encoder of the terminal device is configured to perform the encoding operation, and a bit width depth of an image supported by the encoder is greater than or equal to 8 bits.

[0019] In a second aspect, the present application provides an image processing method applied to a cloud device, the method comprising: obtaining an encoded code stream; decoding a second original RAW image from the encoded code stream, the second RAW image comprising an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; and performing post-processing on the second RAW image to obtain a first RAW image.

[0020] In a possible implementation, the post-processing on the second RAW image to obtain the first RAW image comprises: performing color gamut inverse transformation on the second RAW image to obtain an image in YCoCgDg format; and splicing the images of the four components according to a set splicing manner to obtain the first RAW image.

[0021] In a possible implementation, the color gamut inverse transformation on the second RAW image to obtain an image in YCoCgDg format comprises: performing linear transformation on the second RAW image to obtain a transformed second RAW image; and performing color gamut inverse transformation on the transformed second RAW image to obtain an image in YCoCgDg format.

[0022] In a possible implementation, the decoding of the second original RAW image from the encoded code stream comprises: decoding the second original RAW image from the encoded code stream according to any one of the following encoding standards: H264, H265, H266, and JPEG low latency.

[0023] In a possible implementation, the obtaining of the encoded code stream comprises: receiving the encoded code stream from a terminal device; or obtaining the encoded code stream from a storage medium.

[0024] In a possible implementation, the decoder of the cloud device is configured to perform the decoding operation, and a bit width depth of an image supported by the decoder is greater than or equal to 8 bits.

[0025] In a third aspect, the present application provides an image processing apparatus, comprising: an obtaining module configured to obtain a first RAW image; a preprocessing module configured to preprocess the first RAW image to obtain a second RAW image, the second RAW image comprising an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; an encoding module configured to encode the second RAW image to obtain a first encoded bitstream; and a storage module configured to store the first encoded bitstream, or a sending module configured to send the first encoded bitstream to a cloud device.

[0026] In a possible implementation, the preprocessing module is specifically configured to: perform color gamut transformation on the first RAW image to obtain images of four components in the YCoCgDg format; and splice the images of the four components according to a set splicing manner to obtain the second RAW image. It should be understood that the YCoCgDg format is only an example and is not a limitation. In other embodiments, the second RAW image can also be in other color gamut formats and can be preprocessed in a similar manner to the present application.

[0027] In a possible implementation, the set splicing manner comprises any one of the following: a matrix splicing manner, a horizontal splicing manner, and a vertical splicing manner; and the shape of the second RAW image comprises any one of the following: a checkboard, a horizontal strip, and a vertical strip.

[0028] In a possible implementation, the preprocessing module is specifically configured to: perform nonlinear transformation on the first RAW image to obtain a transformed first RAW image; and perform color gamut transformation on the transformed first RAW image to obtain images of four components in the YCoCgDg format.

[0029] In a possible implementation, the preprocessing module is further configured to: perform desensitization detection and desensitization processing based on the image of the luminance component among the four components to obtain a desensitized image of the luminance component; and splice the desensitized image of the luminance component, the images of the color components, and the image of the difference component among the four components according to the set splicing manner to obtain the second RAW image.

[0030] In a possible implementation, the obtaining module is further configured to: obtain the first encoded code stream from a storage medium; the apparatus further includes: a decoding module configured to decode a second RAW image from the first encoded code stream; a desensitization module configured to perform desensitization detection and desensitization processing based on the second RAW image to obtain a third RAW image; the encoding module is further configured to: encode the third RAW image to obtain a second encoded code stream; and the sending module is further configured to: send the second encoded code stream to the cloud device.

[0031] In a possible implementation, the desensitization module is specifically configured to: perform desensitization detection and desensitization processing based on an image of a luminance component in the second RAW image to obtain a desensitized image of the luminance component; and splice the desensitized image of the luminance component, an image of a color component in the second RAW image, and an image of a difference component in a set splicing manner to obtain the third RAW image.

[0032] In a possible implementation, the encoding module is specifically configured to: encode the second RAW image to obtain the first encoded code stream by using any one of the following encoding standards: an H264 encoding standard, an H265 encoding standard, an H266 encoding standard, and a JPEG low-latency encoding standard.

[0033] In a possible implementation, the terminal device includes an image acquisition module, and the obtaining module is specifically configured to: acquire a RAW image corresponding to a current target scene by using the image acquisition module to obtain the first RAW image; or obtain the first RAW image from a storage medium.

[0034] In a possible implementation, the terminal device further includes an image signal processing (ISP) module, and the ISP module is configured to: process the first RAW image to obtain a fourth image, the fourth image including an image in YUV format, Y being a luminance component, and U and V being color components; encode the fourth image to obtain a third encoded code stream; and store the third encoded code stream or send the third encoded code stream to the cloud device.

[0035] In a possible implementation, an encoder of the terminal device is configured to perform an encoding operation, and the encoder supports a bit width depth of an image being greater than or equal to 8 bits.

[0036] In a fourth aspect, the present application provides an image processing apparatus, comprising: an obtaining module, configured to obtain an encoded code stream; a decoding module, configured to decode a second original RAW image from the encoded code stream, the second RAW image comprising an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; and a post-processing module, configured to post-process the second RAW image to obtain a first RAW image.

[0037] In a possible implementation, the post-processing module is specifically configured to: perform color gamut inverse transformation on the second RAW image to obtain an image in YCoCgDg format; and splice the images of the four components according to a set splicing manner to obtain the first RAW image.

[0038] In a possible implementation, the post-processing module is specifically configured to: perform linear transformation on the second RAW image to obtain a transformed second RAW image; and perform color gamut inverse transformation on the transformed second RAW image to obtain an image in YCoCgDg format.

[0039] In a possible implementation, the decoding module is specifically configured to decode a second original RAW image from the encoded code stream according to any one of the following encoding standards: H264, H265, H266, and JPEG low latency.

[0040] In a possible implementation, the obtaining module is specifically configured to: receive the encoded code stream from a terminal device; or obtain the encoded code stream from a storage medium.

[0041] In a possible implementation, a decoder of the cloud device is configured to perform a decoding operation, and the decoder supports a bit width depth of an image being greater than or equal to 8 bits.

[0042] In a fifth aspect, the present application provides a computing device, comprising a processor coupled with a memory, wherein the processor is configured to execute a computer program or instructions stored in the memory, so that the apparatus performs the method according to the first aspect and any possible implementation of the first aspect, or performs the method according to the second aspect and any possible implementation of the second aspect.

[0043] In a sixth aspect, the present application provides an image processing system, comprising a terminal device and a cloud device, wherein the terminal device is configured to implement the method according to the first aspect and any possible implementation of the first aspect, and the cloud device is configured to implement the method according to the second aspect and any possible implementation of the second aspect.

[0044] In a seventh aspect, the present application provides an image processing system, comprising a transceiver, a memory and a processor; the transceiver is configured to receive and send data; the memory is configured to store computer program instructions and data; and the processor is configured to execute the computer program instructions and data in the memory, so as to enable the image processing system to perform the method according to the first aspect and any possible implementation manner of the first aspect, or perform the method according to the second aspect and any possible implementation manner of the second aspect.

[0045] In an eighth aspect, the present application provides a vehicle, comprising a unit configured to perform the method according to the first aspect and any possible implementation manner of the first aspect.

[0046] In a ninth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer program or instructions, and when the computer program or instructions are executed by a computer, the computer is enabled to perform the method according to the first aspect and any possible implementation manner of the first aspect, or perform the method according to the second aspect and any possible implementation manner of the second aspect.

[0047] In a tenth aspect, the present application provides a computer program product, wherein the computer program product comprises computer program or instructions, and when the computer program or instructions are executed on a computer, the computer is enabled to perform the method according to the first aspect and any possible implementation manner of the first aspect, or perform the method according to the second aspect and any possible implementation manner of the second aspect.

[0048] In an eleventh aspect, the present application provides a chip, comprising a processor coupled with a memory, and the processor is configured to execute computer program or instructions stored in the memory, and when the computer program or instructions are executed, the method according to the first aspect and any possible implementation manner of the first aspect is implemented, or the method according to the second aspect and any possible implementation manner of the second aspect is implemented.

[0049] On the basis of the implementation of the above aspects, the present application can be further combined to provide more implementations.

[0050] The beneficial effects of the above-mentioned second aspect to eleventh aspect can refer to the description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1 is a structural schematic diagram of an end-cloud system according to an embodiment of the present application;

[0052] FIG. 2 is a hardware structural schematic diagram of a vehicle according to an embodiment of the present application;

[0053] FIG. 3 is a flow diagram of an image processing method according to an embodiment of the present application;

[0054] FIGS. 4a-4e are schematic diagrams of RAW images in different formats according to an embodiment of the present application;

[0055] FIGS. 5a-5c are schematic diagrams of different stitching manners according to an embodiment of the present application;

[0056] FIG. 6 is a flow diagram of a RAW image based desensitization detection and desensitization processing according to an embodiment of the present application;

[0057] FIG. 7 is a schematic diagram of a RAW image based desensitization detection and desensitization processing according to an embodiment of the present application;

[0058] FIG. 8 is a flow diagram of another RAW image based desensitization detection and desensitization processing according to an embodiment of the present application;

[0059] FIG. 9a is a schematic diagram of a modular structure of a terminal device according to an embodiment of the present application;

[0060] FIGS. 9b-9c are flow diagrams of an image processing method according to an embodiment of the present application;

[0061] FIG. 10 is a schematic diagram of an image processing apparatus according to an embodiment of the present application;

[0062] FIG. 11 is a schematic diagram of another image processing apparatus according to an embodiment of the present application;

[0063] FIG. 12 is a schematic diagram of another image processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] Before introducing the technical solutions provided by the present application, first, some terms involved in the present application are explained and described so as to facilitate the understanding of the skilled in the art.

[0065] (1) Image encoding and image decoding: image encoding can also be referred to as image compression, which refers to a technology for representing an image or information contained in an image with fewer bits under the condition of meeting certain quality (such as the requirement of signal-to-noise ratio or subjective evaluation score). Image decoding is the reverse process of image encoding.

[0066] (2) Bayer raw image, also known as Bayer image or raw image. Raw means "unprocessed" in its original sense, which can be understood as: the Bayer raw image refers to the original data converted by a charge coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor and the like of a camera from a captured light source signal to a digital signal, that is, the original image inside the camera. Therefore, the Bayer raw image can also be conceptualized as "original image encoding data" or more figuratively as "digital negative".

[0067] In addition, in the embodiment of the present application, the Bayer raw image includes three color components, and each pixel point in the Bayer raw image has only one color component, and the value of the color component can be equivalent to the pixel value of the pixel point. In one example, the three color components are red (R) component, blue (B) component, and green (G) component. In another example, the three color components are R component, B component, and yellow (Y') component. Which color components are included in the Bayer raw image is related to the color filter in the camera.

[0068] (3) RGB color space and YUV color space:

[0069] Generally, an image is composed of the smallest unit of pixels, and each pixel information is also composed of different brightness RGB information, that is, the original image can include information of three components of red R, green G, and blue B. If the RGB signal is directly used for image signal transmission, it cannot be compatible with black and white televisions, and the cost of occupied bandwidth is high, so the traditional image signal processing (ISP) device converts the image from the RGB color space to the YUV color space for transmission.

[0070] In the YUV color space, an image information is divided into one luminance information and two color information, the luminance information is represented by Y, and the color information is composed of hue and saturation, and the hue and saturation are represented by UV. When performing image signal transmission, the analog component video or YUV signal needs to be digitally sampled, that is, the luminance information and the color information need to be sampled. Common YUV sampling methods include YUV444, YUV422, YUV420, YUV411, etc.

[0071] (4) Standard encoding compression technology: traditional standard encoding compression technology, such as H266 / H265 / H264 / JPEG, etc. video / image encoding standard technology, only provides a solution for some color formats (such as RGB format or YUV format).

[0072] In the field of advanced driving assistant system (ADAS), automated driving system (ADS) or intelligent driving system, with the continuous improvement of camera resolution, frame rate, bit width depth, etc., the demand for transmission bandwidth of video image (such as RAW image collected by camera) output is increasing. In order to alleviate the pressure of video image transmission, the method of image encoding (or can be called image compression) is usually used to reduce the demand for transmission bandwidth. In the intelligent driving application scenario, in order to meet the requirement of safety first for automated driving, image compression (such as RAW image compression) needs to meet the compression requirements of low latency, low complexity and high compression performance.

[0073] However, the above-mentioned H266 / H265 / H264 standard encoding compression technology has no large-scale application standard for RAW image coding and decoding. JPEG low latency encoding standard (JPEG Extra Speed, JPEG-XS) (or ResPro) is the first low latency image encoding and decoding international standard for Bayer raw image. Application scenarios include compression of video signals inside the camera, automated driving Bayer raw video compression transmission, high-speed video storage interface compression, etc. Although the compression performance of RAW data can be improved, the compression performance is between 3 times and 6 times, which still cannot meet the high compression performance requirement of high-order intelligent driving application scenario. For example, if the image data at the vehicle end is compressed 120 times by the encoder in the form of YUV420 and uploaded to the cloud device, the compression will greatly reduce the image quality uploaded to the cloud device due to the limitation of transmission bandwidth and vehicle end traffic.

[0074] To solve the above problems, the embodiments of the present application provide an image processing method and device for realizing encoding compression for RAW image, which is suitable for various existing encoding standards and can guarantee the quality of compressed image. The method and device are based on the same technical concept. Since the principles of the method and device for solving problems are similar, the implementation of the device and the method can be mutually referred to, and the repeated parts will not be described again. Moreover, in various embodiments of the present application, the terms and / or descriptions of various embodiments are consistent and can be mutually referred to if there is no special description and logical conflict. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0075] It should be noted that "at least one" in the embodiments of the present application means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0076] In addition, unless otherwise specified, the ordinal numbers mentioned in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the priority or importance of the plurality of objects. For example, the first RAW image and the second RAW image are only used to distinguish different RAW images, and do not represent the difference in priority or importance of the two RAW images.

[0077] The embodiments of the present application can be applied to the scene that the terminal device collects images or videos and uploads to the cloud device, that is, in the end-cloud collaborative scene. Referring to FIG. 1, it is a structural schematic diagram of an end-cloud system provided by the embodiments of the present application. The "end" of the end-cloud collaboration refers to the terminal device, and the "cloud" refers to the cloud device. The cloud device can also be a cloud server or a cloud platform. The cloud device can have the function of massive computing power. The end-cloud system can include a terminal device 110 and a cloud device 120, and the terminal device 110 can be connected with the cloud device 120 through a wireless network.

[0078] In an embodiment, the cloud device 120 can be a computer server or a server cluster composed of multiple servers, and the implementation architecture of the cloud device 120 is not limited in the present application. The terminal device 110 can be a device with a photographing function and a network function. The terminal device 110 can also have a computing processing function. The terminal device 110 can be a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or the like, or can also be a professional photographing device such as a digital camera, a single-lens reflex camera / micro-single camera, a sports video camera, a gimbal camera, a drone, and the like, and the specific type of the terminal device is not limited in the embodiments of the present application. The number of terminal devices 110 included in the terminal-cloud system can be one or multiple. The types of the multiple terminal devices 110 can be the same or different.

[0079] In an example, taking a vehicle implemented by the terminal device 110 as an example, the hardware structure of the vehicle is introduced. Referring to FIG. 2, a schematic diagram of a hardware structure of a vehicle is shown.

[0080] The vehicle can include a processor 210, an external memory interface 220, an internal memory 221, an automotive bus interface 230, a communication module 240, a sensing system 250, a display screen 260, etc. Among them, the automotive bus interface 230 can include but is not limited to a controller area network (CAN) bus interface, a FlexRay bus interface, a LIN bus interface, etc. Through various bus interfaces, the interconnection between various components inside the vehicle can be realized, and the interconnection between the vehicle and peripheral devices can also be realized. The vehicle can also communicate with servers or other vehicles through the communication module 240 and the network. The sensing system 250 can include at least one sensor, and the processor can identify the environment or scene in which the vehicle is located through the sensing information provided by the at least one sensor, to assist the vehicle to realize automatic driving or intelligent auxiliary driving function. Illustratively, the sensing system 250 can include but is not limited to at least one of an image sensor (such as a camera), a light sensor, a distance sensor, a light detection and ranging (LIDAR), a millimeter-wave radar (RADAR), etc. The number of different types of sensors can be one or more, and the deployment position can be inside the vehicle or outside the vehicle.

[0081] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device 110. In other embodiments of the present application, the terminal device can include more or fewer components than the illustration, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0082] In specific implementation, the processor 210 can be deployed on a related vehicle-mounted device of the vehicle, for example, deployed in a mobile data center (MDC) or a cockpit domain controller (CDC) of the vehicle, or a vehicle control unit (VCU), or a vehicle domain controller (VDC), or an advanced driver assistance system (ADAS) domain controller, or a control unit of other components of the vehicle. The product form and deployment manner of the processor are not limited in the embodiments of the present application.

[0083] The processor 210 can include one or more processing units, for example: the processor 210 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0084] The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0085] The processor 210 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory can save instructions or data that the processor 210 has just used or repeatedly uses. If the processor 210 needs to use the instructions or data again, it can directly call from the memory. Avoiding repeated access, reducing the waiting time of the processor 210, thus improving the efficiency of the system.

[0086] In one example, the terminal device can realize image shooting function and processing of image through at least one of camera, ISP, video codec, GPU, display screen 260 and application processor, etc.

[0087] For example, the camera can be used to capture still images or videos. An object projects an optical image through a lens to a photosensitive element. The photosensitive element can be a CCD or CMOS phototransistor. The photosensitive element converts the optical signal to an electrical signal, which is then passed to an ISP in the processor 210 to convert to a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal to a standard RGB, YUV, etc. format image signal. In some embodiments, the processor 210 can trigger the camera to capture at least one image according to a program or instructions in the internal memory 221, and process the at least one image according to the program or instructions, such as image post-processing (e.g., skin beautification, super resolution processing to enhance clarity, etc.). The processed image can be displayed by the display 260. In some embodiments, the vehicle can include one or N2 cameras, where N2 is a positive integer greater than 1. For example, the vehicle can include a front-facing camera, a rear-facing camera, or a 360-degree surround view camera. The vehicle can also include a cabin camera, for example.

[0088] The ISP in the processor 210 can be used to process data fed back by the image sensor. For example, when taking a picture, the shutter is opened, light passes through the lens to the camera photosensitive element, the optical signal is converted to an electrical signal, and the camera photosensitive element then performs analogue-to-digital (A / D) conversion on the electrical signal to output a corresponding digital signal. The digital signal is passed to the ISP for processing to convert to a visible image. The digital signal output by the camera sensor to the ISP can be understood as the original image captured by the camera, i.e. a RAW image. The ISP can perform ISP processing on the RAW image to ultimately generate a YUV image.

[0089] Exemplarily, the ISP processing can include: black frame correction, bad pixel correction (DPC), RAW domain noise reduction, black level correction (BLC), lens shading correction (LSC), auto white balance (AWB) gain, green balance correction, demosaic color interpolation, color correction matrix (CCM), dynamic range compression (DRC), gamma, 3D look up table (LUT), YUV domain noise reduction, sharpening, detail enhancement, etc. The ISP can also optimize the exposure, color temperature, etc. of the shooting scene.

[0090] In some embodiments, part of the functions of the ISP can be arranged in the image sensor, and other ISP processing can be retained in the ISP device of the processor. For example, the black frame correction, bad pixel correction, RAW domain noise reduction, lens shading correction, black level correction, auto white balance gain, green balance correction, DRC, etc. in the above-mentioned ISP processing can be arranged in the camera or other sensor with camera function. Optionally, the function of the ISP for optimizing the exposure, color temperature, etc. of the shooting scene can also be integrated in the camera end. The demosaic color interpolation, color correction, gamma, 3D look up table, YUV domain noise reduction, sharpening, detail enhancement, etc. in the above-mentioned ISP processing can be arranged in the processor.

[0091] The digital signal processor is used to process digital signals, which can process not only digital image signals but also other digital signals. For example, when the terminal device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0092] The video codec is used to compress or decompress digital video. The vehicle can support one or more video codecs. In this way, the vehicle can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0093] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also continuously self-learn. Through the NPU, intelligent cognition and other applications of a terminal device can be implemented, for example, image recognition, face recognition, speech recognition, text understanding, and the like.

[0094] The external memory interface 220 can be used to connect an external memory card, for example, a Micro SD card, to realize the expansion of the storage capacity of the terminal device. The external memory card communicates with the processor 210 through the external memory interface 220 to realize a data storage function. For example, music, video, and the like are saved in the external memory card.

[0095] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program (such as a camera application) required by a function, and the like. The data storage area can store data (such as images captured by a camera) created during use of the terminal device, and the like. In addition, the internal memory 221 can include a high-speed random access memory and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 210 executes various function applications and data processing of the terminal device by running instructions stored in the internal memory 221 and / or instructions stored in a memory disposed in the processor.

[0096] It can be understood that the structure shown in FIG. 2 does not constitute a specific limitation on the terminal device. In some embodiments, the terminal device can also include more or fewer components than those shown in FIG. 2, or combine certain components, or split certain components, or different component arrangements, and the like. Alternatively, some components shown in FIG. 2 can be implemented in hardware, software, or a combination of software and hardware.

[0097] In addition, when the terminal device is another tablet computer, wearable device, AR / VR device, notebook computer, UMPC, netbook, PDA, and the like mobile terminal, or a digital camera, single-lens reflex camera / micro-single camera, action camera, gimbal camera, unmanned aerial vehicle, and the like professional shooting device, the specific structure of these other terminal devices can also be referred to that shown in FIG. 2. Exemplarily, the other terminal device can increase or reduce components on the basis of the structure given in FIG. 2, which will not be described one by one here.

[0098] It should also be understood that one or more photographing applications can be run in the vehicle, so that the photographing function is realized by running the photographing application. For example, the photographing application can include a system-level application "camera" application. For another example, the photographing application can also include other applications capable of photographing installed in the terminal device.

[0099] After the vehicle obtains the RAW image through the camera, the vehicle can process the RAW image to obtain a corresponding encoding code stream, and implement a corresponding visual task based on the encoding code stream.

[0100] Taking the terminal device implementation as an example, on the one hand, the vehicle can perform a series of ISP processing on the RAW image, and then transmit the related image data to the automatic driving system or the intelligent auxiliary driving system of the vehicle, so as to realize the automatic driving function or the intelligent driving auxiliary function of the vehicle. On the other hand, the vehicle can process the RAW image to obtain a corresponding encoding code stream, which can be stored in the local storage medium of the vehicle, or can be sent to the cloud device. The cloud device can receive the encoding code stream from the terminal device, and perform decoding on the encoding code stream to restore the original RAW image, and then perform post-processing to execute related visual tasks. For example, in the cloud photographing scenario, the powerful computing power of the cloud device is used for image processing, which can solve the problem that the processing performance and limited memory resources of the terminal device limit the image processing effect, and better image processing effect can be obtained.

[0101] For ease of understanding, the following will be described in detail in combination with a method flowchart.

[0102] As shown in FIG. 3, the method can include the following steps:

[0103] S310: The terminal device obtains a first original RAW image.

[0104] In the embodiment of the present application, the terminal device can include an image acquisition module and a preprocessing module. The image acquisition module can be, for example, the camera in FIG. 2, and the preprocessing module can be, for example, a module integrated in the processor 210 in FIG. 2. When S310 is implemented, the preprocessing module of the terminal device can obtain a first RAW image by real-time acquisition of the RAW image corresponding to the current target scene through the image acquisition module. Alternatively, the preprocessing module can obtain the first RAW image provided by the image acquisition module from the storage medium.

[0105] In one example, when the image capture module does not have ISP function, the first RAW image is a RAW image captured by the image capture module on the vehicle, for example, a RAW image captured based on a Bayer pattern. For example, taking vehicle A as an example, when the image capture module located on vehicle A captures an environmental information to form a RAW image, the RAW image can be sent to the pre-processing module. Alternatively, the image capture module can also cache the RAW image. When the image capture module caches the RAW image, the image capture module can send the cached RAW image to the pre-processing module in response to an image request from the pre-processing module.

[0106] In another example, the image capture module can include partial ISP function, and the first RAW image is a RAW image obtained by processing a RAW image captured by the image capture module. For example, taking vehicle A as an example, when the image capture module located on vehicle A captures an environmental information to form a RAW image, the RAW image can be sent to the ISP module in the image capture module. After receiving the RAW image, the internal ISP module can process and optimize the RAW image to obtain the first RAW image.

[0107] In another example, if the image capture module does not have ISP function, the first RAW image can be a RAW image obtained by processing a RAW image captured by the image capture module on the vehicle. For example, taking vehicle A as an example, when the image capture module located on vehicle A captures an environmental information to form a RAW image, the RAW image can be sent to the ISP module outside the image capture module. After receiving the RAW image, the external ISP module can process and optimize the RAW image to obtain the first RAW image.

[0108] The internal ISP module or the external ISP module can perform at least one of the following on the original RAW image to obtain the first RAW image: black frame correction, bad pixel correction, RAW domain noise reduction, lens brightness correction, black level correction, automatic white balance gain, green balance correction, DRC, etc. The internal ISP module or the external ISP module can send the first RAW image to the preprocessing module, or can cache the first RAW image. When the internal ISP module or the external ISP module caches the first RAW image, the cached first RAW image can be sent to the preprocessing module in response to an image request of the preprocessing module. For ease of distinction, the above-mentioned ISP processing on the original RAW image can also be referred to as first ISP. Subsequently, the terminal device can further perform other ISP processing on the first RAW image. For example, the external ISP module can perform demosaicing color interpolation, color correction, gamma, 3D lookup table, YUV domain noise reduction, sharpening, and detail enhancement on the first RAW image.

[0109] S320: The terminal device pre-processes the first RAW image to obtain a second RAW image.

[0110] In an embodiment of the present application, the Bayer raw format of the first RAW image can include, but is not limited to, RGGB format or GRGB format, etc. The image signals of the four channels are different from the traditional 3-channel RGB format or YUV format image signals, and are not suitable for direct encoding using a traditional encoder. Further processing is required before encoding. The further processing is referred to as preprocessing of the first RAW image in the embodiment of the present application, and the RAW image obtained after preprocessing is referred to as a second RAW image.

[0111] Exemplarily, the second RAW image can include a YCoCgDg format image, Y is a luminance component, Co and Cg are color components, and Dg is a difference component. The YCoCgDg format image can be obtained by directly performing color domain conversion on the Bayer raw format image, without converting the first RAW image to RGB format by demosaicing color interpolation, etc.

[0112] In one example, when S320 is implemented, the preprocessing module of the terminal device can directly perform color gamut conversion on the first RAW image to obtain an image of four components in YCoCgDg format; and splice the image of four components according to a set splicing manner to obtain the second RAW image. In another example, when S320 is implemented, the preprocessing module of the terminal device can also perform nonlinear transformation on the first RAW image to obtain a transformed first RAW image, then perform color gamut conversion on the transformed first RAW image to obtain an image of four components in YCoCgDg format, and splice the image of four components according to a set splicing manner to obtain the second RAW image.

[0113] By performing nonlinear transformation on the first RAW image, the RAW image data distribution can be more suitable for subsequent encoding processing, so that the image encoding performance can be improved, and the image encoding delay can be reduced. By performing color gamut conversion on the first RAW image (including the first RAW image after nonlinear transformation), the correlation between different channel data can be removed, and the redundant information between color signals can be removed, the image compression performance can be improved, and the computational complexity (or image processing complexity) can be reduced, for example, the computational complexity can be saved by more than 60%, so as to improve the image compression performance. The second RAW image obtained by splicing the image of four components according to the set splicing manner is suitable for the YUV format supported by the encoder, and can directly use the existing encoding standard.

[0114] When performing nonlinear transformation on the first RAW image, the bit depth of the first RAW image can be 24 bits, and the basic principle of nonlinear transformation on the first RAW image can be to map the 24-bit RAW data to the human eye sensitivity domain of the selected bit depth (referred to as bit depth). The basic operation of the above nonlinear transformation can include the following two kinds:

[0115] (1) Map the 24-bit RAW data to 16 / 12 / 10 / 8-bit bit depth; (2) Gamma-like transformation, map the original RAW data to the human eye sensitivity domain through nonlinear gamma transformation, and the transformation form can be a function, a lookup table, a key position point, etc.

[0116] Taking the function-based nonlinear transformation as an example, the preprocessing module can perform nonlinear transformation on the first RAW image through the following expression (1).

[0117] Wherein, X is used to represent the first normalized pixel value of a certain pixel point in the first RAW image, Y is used to represent the second normalized pixel value (i.e. the first normalized pixel value after the non-linear transformation) of the pixel point, and the value of gamma can be any suitable value. Optionally, the pre-processing module can also use other non-linear transformation methods to realize the non-linear transformation of the plurality of pixel points included in the first RAW image, which will not be described here.

[0118] It should be understood that in the embodiments of the present application, the bit depth of the original RAW image obtained by the image acquisition module can be 24 bits, and the RAW image data of 24 bits is too large, and the transmission pressure is also large. Before being transmitted to the pre-processing module via a wired cable, the related device (including the image acquisition module integrated with the first ISP function or an external ISP module) can also use the Pixel Wise Linear (PWL) compression method to compress the original RAW image, such as compressing the RAW data from 24 bits to 12 bits, reducing the transmission bandwidth requirement by 1 times. After the data is transmitted to the pre-processing module, the 12-bit compressed data is restored to 24 bits by the pre-processing module for PWL decompression, and the first RAW is obtained.

[0119] When the color gamut transformation is performed on the first RAW image (including the first RAW image after the non-linear transformation), the pre-processing module can extract the RGGB four components of the RAW image in the Bayer format to the four-channel image in the RGGB format or the GRGB format, and then use the color gamut transformation matrix to transform the RAW image in the RGGB format or the GRGB format into the RAW image in the YCoCgDg format.

[0120] For example, as shown in FIG. 4a, when the Bayer raw format of the first RAW image is RGGB format, the color component of the pixel point in the first row and the first column of the first RAW image is R, the color component of the pixel point in the first row and the second column is G, the color component of the pixel point in the second row and the first column is G, and the color component of the pixel point in the second row and the second column is B, and the four color components of the pixel points are cyclically present. For example, as shown in FIG. 4b, when the Bayer raw format of the first RAW image is GRGB format, the color component of the pixel point in the first row and the first column of the first RAW image is G, the color component of the pixel point in the first row and the second column is R, the color component of the pixel point in the second row and the first column is G, and the color component of the pixel point in the second row and the second column is B, and the four color components of the pixel points are cyclically present.

[0121] Taking one image unit (such as image unit A) included in the first RAW image as an example, the first value is a Y value, the second value is a Dg value, the third value is a Co value, and the fourth value is a Cg value. Assuming that the pixel value (or color component) of pixel point 1 included in image unit A is G1, the pixel value of pixel point 2 is G2, the pixel value of pixel point 3 is R, and the pixel value of pixel point 4 is B. The pre-processing module can calculate the Y value, Dg value, Co value, and Cg value corresponding to image unit A through the following expression (2):

[0122] As shown in FIG. 4c, the color component of the pixel point in the first row and the first column of the second RAW image is Y, the color component of the pixel point in the first row and the second column is Dg, the color component of the pixel point in the second row and the first column is Co, and the color component of the pixel point in the second row and the second column is Cg, and the pixel points of the four color components are cyclically present.

[0123] It should be understood that the pixel value G1 of pixel point 1, the pixel value G2 of pixel point 2, the pixel value R of pixel point 3, and the pixel value B of pixel point 4 are all pixel values after nonlinear transformation. The color gamut conversion process based on expression (2) is relatively simple, generally only involving addition and shifting. Alternatively, the pre-processing module can also use the existing color gamut conversion method or other color gamut conversion method to realize the color gamut conversion of the first RAW image after nonlinear transformation. The above color gamut conversion can make the encoding of the RAW domain data more suitable for the existing encoding standard, and can maximize the reduction of the loss of the encoded compressed image quality caused by the inadaptation of the image format.

[0124] In another example, when the first RAW image (including the first RAW image after nonlinear transformation) is subjected to color gamut conversion, a Star Tetrix transform method can also be used to obtain a four-channel image by referring to the pixels of the first four rows. The four-channel image can be represented as a YCrCr△ (Delta) format image, and the second RAW image can include a YCbCr△ format image. The YCbCr△ format is a different name of the YCoCgDg format, and the essence is still a YCoCgDg format image.

[0125] In specific implementation, the pre-processing module can generate color components Cb and Cr by predicting R and B from four green components Gx (x={l, r, t, b}) around the target pixel point in the first RAW image, as shown in FIG. 4d. The prediction process can satisfy the following expression (3):

[0126] where l, r, t and b represent the sample positions of the current pixel left, right, top and bottom, respectively.

[0127] Then, the pre-processing module can generate the luminance components Y1 and Y2 by updating G from the four surrounding and (x={l, r, t, b}). The process can satisfy the following expression (4):

[0128] Then, the pre-processing module can generate the luminance difference component △ by predicting Y1 from the four surrounding Y2, satisfying the following expression (5):

[0129] Then, the pre-processing module can generate the final luminance component Y by updating Y2 from the four surrounding, satisfying the following expression (6):

[0130] After the Star Tetrix transform described above, a four-channel image can be obtained as shown in FIG. 4e, including an image of the Y component, an image of the Cr component, an image of the Cb component, and an image of the Delta (△) component. The Y component is a luminance component, the Cr and Cb are color components, and the Delta is a difference component.

[0131] For the above-described setting of the splicing mode, the exemplary setting of the splicing mode can include any one of the following: a matrix splicing mode, a horizontal splicing mode, and a vertical splicing mode.

[0132] The matrix splicing mode refers to splicing the four-component images in the YCoCgDg format according to a 2*2 matrix format, and the shape of the second RAW image obtained after splicing is a grid, as shown in FIG. 5a. The horizontal splicing mode or the vertical splicing mode can also be referred to as a vector splicing mode, which refers to splicing the four-component images in the YCoCgDg format according to a 1*4 row vector or a 4*1 column vector format, and the shape of the second RAW image obtained after splicing is a horizontal strip or a vertical strip, as shown in FIGS. 5b and 5c.

[0133] It is worth noting that in the above matrix splicing manner, the length and width of the YCoCgDg image obtained after splicing are consistent with those of the first RAW image, but 0.5 frames of additional time delay (about 16 ms @ 30 fps) can be introduced. In addition, from the perspective of hardware implementation, the image width after splicing is only about half of that after splicing based on the vector splicing manner, so the matrix splicing manner requires half of the on-chip line buffer (Line Buffer), thereby helping to reduce the chip cost. The vector splicing manner only introduces a 64-pixel row time delay (H265, LCU_size = 64, about 1 ms @ 30 fps), and considering that the traditional image encoding scheme (or traditional image coding and decoding scheme) also has this time delay, the vector splicing manner actually does not introduce additional time delay. In addition, from the perspective of image encoding performance (or image compression performance), since the image content at the splicing boundary is basically irrelevant, the above matrix splicing manner and the vector splicing manner have almost the same compression performance. Furthermore, compared with the above matrix splicing manner, the vector splicing manner is more suitable for low-latency application scenarios and does not reduce the image compression performance. Therefore, in the specific embodiments, the terminal device can set the splicing manner to be any one of the above matrix splicing manner, horizontal splicing manner, and vertical splicing manner according to the scene needs, which will not be described herein again.

[0134] S330: The terminal device encodes the second RAW image to obtain a first encoded bitstream.

[0135] In the embodiments of the present application, the terminal device can include an encoding module (or encoder, CODEC), which can be, for example, a video CODEC integrated in the processor 210 in FIG. 2. The encoding module can perform the encoding operation of S330, and the bit width depth of the image supported by the encoding module is greater than or equal to 8 bits.

[0136] For example, the encoding module can implement the encoding of the second RAW image according to a set video encoding standard or multiplexing an existing CODEC standard.

[0137] For example, the encoding module can encode the second RAW image according to any one of the following encoding standards to obtain the first encoded code stream: H264 encoding standard, H265 encoding standard, H266 encoding standard, and JPEG low-latency encoding standard. Alternatively, after the series of processes of S310 and S320, an image adapted to an existing RGB format or YUV format encoding standard has been obtained, and therefore, other encoding standards (such as AV1 or AVS3, etc.) can also be used to encode the second RAW image, and the embodiments of the present application do not make any limitation in this regard. In a specific implementation, a suitable encoding and decoding standard can be selected according to different application scenarios to obtain better compression performance, and at the same time, the openness and flexibility of the scheme are improved, and details are not described herein.

[0138] S340: The terminal device stores the first encoded code stream, or sends the first encoded code stream to the cloud device. Correspondingly, the cloud device can receive the first encoded code stream from the terminal device.

[0139] In the embodiments of the present application, the terminal device can store the first encoded code stream in a local non-volatile storage medium, and the type of the non-volatile storage medium is not limited in the embodiments of the present application.

[0140] The cloud device can be the cloud server or the cloud platform introduced in the foregoing. The cloud device can decode the first encoded code stream according to a set video encoding standard or by multiplexing an existing encoding and decoding standard, decode the second RAW image from the first encoded code stream, and then the cloud device can perform post-processing on the second RAW image to obtain the first RAW image.

[0141] For example, the second RAW image includes an image in YCoCgDg format, Y is a luminance component, Co and Cg are color components, and Dg is a difference component. The post-processing can include color domain inverse transformation processing and linear transformation. The cloud device can directly perform color domain inverse transformation on the second RAW image to obtain an image in YCoCgDg format, and splice the images of the four components according to a set splicing manner to obtain the first RAW image. Alternatively, the cloud device can perform linear transformation on the second RAW image to obtain a transformed second RAW image, perform color domain inverse transformation on the transformed second RAW image to obtain an image in YCoCgDg format, and then splice the images of the four components according to a set splicing manner to obtain the first RAW image.

[0142] The cloud device can transmit the first RAW image to a downstream task module. The downstream task module can be a module including a neural network model based on a RAW domain, or a module including a machine vision model or a human eye vision model based on an RGB format. The cloud device can directly transmit the first RAW image to the downstream task module including the neural network model based on the RAW domain. The cloud device can transmit an image in an RGB format obtained after performing a complex ISP process (for example, the second ISP described above) on the first RAW image to the module including the machine vision model or the human eye vision model based on the RGB format.

[0143] In an optional embodiment, the terminal device described above can also support an image encoding and compression capability for an RGB format or a YUV format. In this scheme, the terminal device can include an ISP module, which can perform a complex ISP process on the first RAW image to obtain an image in an RGB format or a YUV format suitable for human eyes. For example, the terminal device can perform a process on the first RAW image through the ISP module to obtain a fourth image including an image in a YUV format, where Y is a luminance component and U and V are color components. The encoding module of the terminal device can also encode the fourth image to obtain a third encoding bitstream. The terminal device can store the third encoding bitstream or send the third encoding bitstream to the cloud device. The encoding module used to encode the fourth image can be the same as the encoding module used to encode the second RAW image in S330, that is, the encoding module of the embodiment of the present application can support both encoding and processing of a RAW image and encoding and processing of an image in an RGB format or a YUV format, thereby realizing multiplexing of existing encoding and decoding standards.

[0144] Thus, through the above method, by performing a preprocessing process on a RAW image suitable for an existing video / image encoding standard, the existing H266 / 265 / 264 / JPEG video / image encoding standard can be multiplexed, and a RAW domain compression technology with engineering usability and rapid commercialization can be maximally realized. Only the input of a hardware encoder needs to be modified, and a RAW domain-oriented encoding standard and a hardware encoder do not need to be redesigned. Meanwhile, the image processing method can consume less traffic under the condition of the same image quality, and can guarantee better image compression and transmission quality under the condition of traffic occupation.

[0145] With the rapid development of intelligent driving technology and automatic driving technology, in some scenarios, intelligent driving vehicles will need to collect external environment information of the road where the vehicle is located and various data of the vehicle itself, for analyzing the surrounding road conditions of the intelligent driving vehicle and the running state of the intelligent driving vehicle, or for training of artificial intelligence models and map customization, etc. Among them, the intelligent driving vehicle usually collects images or videos through a camera device, and obtains the required data through analysis of the images or videos. If the image or video frame contains sensitive information, such as face, license plate, and other private information, desensitization processing needs to be performed on the image or video frame.

[0146] Current desensitization methods are all performed for the input of a desensitization model of an image encoded in YUV format or RGB format. With the development of end-to-end automatic driving and deep image signal processor (DeepISP) technology based on deep learning models, the collection and model training based on raw image files (such as RAW images) are becoming increasingly important, but there is a lack of desensitization detection and desensitization processing technology for RAW images.

[0147] Therefore, in an optional embodiment of the present application, desensitization detection and desensitization processing can also be performed in the method flow shown in FIG. 3. In one example, desensitization detection and desensitization processing can be performed in the process of preprocessing the first RAW image, to reduce the secondary loss of image quality caused by secondary encoding compression. In another example, desensitization detection and desensitization processing can be performed when sending the first encoded code stream to the cloud device, to avoid additional overhead of hardware computing resources caused by real-time desensitization. The following will be introduced respectively in combination with the method flow chart.

[0148] As shown in FIG. 6, when the preprocessing module of the terminal device performs desensitization detection and desensitization processing in the process of S320, the following steps can be included:

[0149] S610 (optional step): The preprocessing module performs nonlinear transformation on the first RAW image to obtain a transformed first RAW image. For detailed implementation details, please refer to the related introduction of S320, which will not be described here.

[0150] S620: The preprocessing module performs color gamut transformation on the transformed first RAW image to obtain an image of four components in YCoCgDg format. For detailed implementation details, please refer to the related introduction of S320, which will not be described here.

[0151] S630: The preprocessing module performs desensitization detection and desensitization processing based on the image of the luminance component in the four components to obtain a desensitized luminance component image.

[0152] S640: The pre-processing module splices the image of the desensitized luminance component, the image of the color component in the four components, and the image of the difference component according to the set splicing mode to obtain a second RAW image.

[0153] For example, a desensitization model can be built in the pre-processing module, which can be trained based on an image in YUV format or an image in RGB format. The pre-processing module can input the image of the luminance component into three input channels corresponding to Y, U, and V or three input channels corresponding to RGB of the desensitization model, respectively, to realize desensitization detection and desensitization processing of the image of the luminance component by using the desensitization model, and output the desensitized image of the luminance component. This method can be considered as copying and inputting the image of the Y channel three times into the desensitization model.

[0154] As shown in FIG. 7, taking a desensitization model supporting 8 bits trained based on an RGB format as an example, the image of the 8-bit Y component is corresponded to three input channels of RGB, which are represented as Y(R), Y(G), and Y(B), respectively. These component images are 8-bit RGB images in the desensitization model, and the desensitized image of the luminance component can be output after desensitization detection and desensitization processing in the desensitization model. When S640 is implemented, the pre-processing module can splice the desensitized image of the luminance component as the image of the Y component in FIGS. 5a-5c with the images of other components according to the set splicing mode to obtain a second RAW image.

[0155] Optionally, the bit width depth of the image supported by the encoder of the terminal device and the desensitization model can be greater than or equal to 8 bits. If the bit width depth of the first RAW image is greater than 8 bits, the pre-processing module can extract the upper 8 bits or the lower 8 bits of the image of the luminance component and input them into three input channels corresponding to Y, U, and V or three input channels corresponding to RGB of the desensitization model, respectively, to realize desensitization detection and desensitization processing of the image of the luminance component by using the desensitization model, and output the desensitized image of the luminance component.

[0156] As shown in FIG. 8, the terminal device can perform desensitization detection and desensitization processing by the processor of the terminal device when sending the first encoded code stream to the cloud device. The process can include the following steps:

[0157] S810: The processor obtains the first encoded code stream from the storage medium.

[0158] The storage medium may be, for example, a non-volatile storage medium local to the terminal device. The first encoded code stream may be an encoded code stream that is pre-processed and encoded by the pre-processing module according to the flow shown in FIG. 3 and stored in the storage medium. Alternatively, the pre-processing module may also obtain the first encoded code stream from other storage medium accessible to the terminal device, and the pre-processing process before encoding is similar to the process described in FIG. 3, which will not be described here. In a specific implementation, the pre-processing module may be implemented, for example, after receiving the cloud uploading instruction from the processor, S810. The present application does not make specific limitations on the triggering manner.

[0159] S820: The processor decodes the second RAW image from the first encoded code stream.

[0160] Exemplarily, the second RAW image includes an image in YCoCgDg format, Y is a luminance component, Co and Cg are color components, and Dg is a difference component. The pre-processing module may be, for example, configured to decode the second RAW image from the first encoded code stream using any one of the following encoding standards: H264 encoding standard, H265 encoding standard, H266 encoding standard, and JPEG low-latency encoding standard. Alternatively, these encoding standards may be replaced by other standards, as long as the encoding end and the decoding end agree.

[0161] S830: The processor performs desensitization detection and desensitization processing based on the second RAW image to obtain a third RAW image. The detailed implementation details are the same as those described above in connection with the method of FIGS. 6-7, and reference can be made to the related description above, which will not be described here.

[0162] S840: The processor encodes the third RAW image to obtain a second encoded code stream. The encoding manner is the same as that described above in connection with S330, and reference can be made to the related description above, which will not be described here.

[0163] S850: The processor sends the second encoded code stream to the cloud device. For example, the pre-processing module sends the second encoded code stream to the cloud device using wireless communication technology through the transceiving antenna of the terminal device.

[0164] Correspondingly, the cloud device can receive the second encoded code stream from the terminal device. Moreover, the cloud device can decode the second encoded code stream to obtain a third RAW image, and then perform post-processing on the third RAW image to obtain a desensitized first RAW image. The cloud device can transmit the desensitized first RAW image to a downstream task module to implement a related visual task, which will not be described here.

[0165] Therefore, by the method of desensitization detection and desensitization processing based on the RAW image shown in FIG. 6 and FIG. 8, the execution timing of the desensitization detection and desensitization processing of the RAW image can be set as needed to protect user privacy and security.

[0166] For ease of understanding, the implementation details of the above image processing method are exemplarily described below in combination with the modular structure of the terminal device.

[0167] As shown in FIG. 9a, in one example, the vehicle can include a camera, an MDC, an encoding module 1, a storage module 1, a decoding module 1, a desensitization module 1, an encoding module 2, and a communication module 1. The camera can be based on the image acquisition module introduced in FIG. 3. The preprocessing module based on FIG. 3 can be integrated in the MDC and cooperates with other modules to realize the RAW domain processing flow involved in the image processing method of the embodiments of the present application.

[0168] As shown in FIG. 9b, the method may, for example, include the following steps:

[0169] S901: After triggering the camera shooting or video shooting function on the vehicle side, the camera can acquire an image or a video frame and send the acquired image or video frame to the MDC. Optionally, part of the ISP function (such as the first ISP processing introduced above) can be integrated in the camera to realize simple processing of the original RAW image, such as the first ISP processing introduced above.

[0170] S902: The deserializer in the MDC can receive the image or video frame from the camera. Wherein, the camera may, for example, send a serial data stream to the MDC in the form of a media stream. The deserializer can convert the serial data stream into a parallel data stream for subsequent processing by the preprocessing module or the ISP module 1 or other modules.

[0171] S903: The preprocessing module can obtain the RAW image through the deserializer, take the RAW image as a first RAW image, and pre-process the first RAW image to obtain a second RAW image. The preprocessing module can send the second RAW image to the encoding module 1.

[0172] As an example, the second RAW image may, for example, be a YCoCgDg format image, Y is a luminance component, Co and Cg are color components, and Dg is a difference component. Exemplarily, the preprocessing steps performed by the preprocessing module can include at least one of nonlinear transformation processing, color domain conversion processing, and re-stitching processing on the first RAW image. For details, refer to the related description in the foregoing combination with FIGS. 3-5c, which will not be repeated here.

[0173] S904: The encoding module 1 can encode the second RAW image to obtain a first encoded code stream.

[0174] S905: The encoding module 1 can send the first encoded code stream to a cloud device through a communication module. Alternatively, the encoding module 1 can store the first encoded code stream through a storage module 1.

[0175] S906: If necessary, for example, when receiving a cloud uploading instruction, the first encoded code stream can be obtained from the storage module 1 by the decoding module 1, and the second RAW image can be decoded from the first encoded code stream. The decoding module 1 can send the decoded second RAW image to the desensitization module 1.

[0176] S907: The desensitization module 1 can perform desensitization detection and desensitization processing based on the decoded second RAW image to obtain a third RAW image. The desensitization module can send the third RAW image to the encoding module 2.

[0177] S908: The encoding module 2 can encode the third RAW image to obtain a second encoded code stream. The encoding module 2 can send the second encoded code stream to a cloud device through the communication module 1. For details of S904-S908, please refer to the related description of FIGS. 3-8, which will not be repeated here.

[0178] It should be understood that the desensitization module 1 is only an example and not any limitation. In other embodiments, the desensitization module can be built into a preprocessing module, and the desensitization detection and desensitization processing of the RAW image can be realized in the preprocessing process according to the method shown in FIG. 6, which will not be repeated here.

[0179] In another example, the MDC of the vehicle can also include an ISP module 1, a security automation (SA) module, a distortion correction module involved in vision preprocessing (VPC), a stitching module, an image preprocessing module, etc. The ISP module 1 can cooperate with other modules to realize the process of RGB domain or YUV domain processing involved in the image processing method of the present application.

[0180] As shown in FIG. 9c, the method can include the following steps, for example:

[0181] S911: After triggering the photographing or video shooting function of the camera on the vehicle side, the camera can collect image or video frames and send the collected image or video frames to the MDC. Optionally, part of the ISP function (for example, the first ISP processing introduced above) can be integrated in the camera to realize simple processing of the original RAW image, for example, the first ISP processing introduced above.

[0182] S912: The deserializer in the MDC can receive the image or video frame from the camera. Wherein, the camera can send a serial data stream to the MDC in the form of a media stream, for example. The deserializer can convert the serial data stream into a parallel data stream for subsequent processing by the pre-processing module or the ISP module 1 or other modules.

[0183] S913: The ISP module 1 can obtain the RAW image through the deserializer and perform ISP processing on the obtained RAW image to convert the RAW image into an image in RGB format suitable for human eyes. Optionally, the ISP module 1 can also convert the image in RGB format into YUV format and sample in YUV420 mode (or other sampling modes).

[0184] S914: The ISP module 1 can provide the YUV format image data obtained after sampling to the automatic driving system / intelligent driving assistance system of the vehicle, so that the automatic driving system / intelligent driving assistance system implements the automatic driving function or the auxiliary driving function of the vehicle based on the YUV format image.

[0185] For example, the ISP module 1 can implement the vehicle parking function after the SA processing and VPC splicing processing of the YUV format image data obtained after sampling.

[0186] Or for example, the ISP module 1 can perform fusion perception processing after the SA processing, VPC distortion correction, and VPC image preprocessing of the YUV format image data obtained after sampling, to implement the automatic driving function or the auxiliary driving function of the vehicle.

[0187] S915: The ISP module 1 can send the YUV format image data obtained after sampling to the encoding module 3 after SA processing and VPC distortion correction processing. The encoding module 3 can encode the YUV format data using existing encoding standards to obtain a third encoding code stream.

[0188] S916: The encoding module 3 can store the third encoding code stream through the storage module 2, or send the third encoding code stream to the cloud device.

[0189] S917: If necessary, the decoding module 2 can obtain the third encoding code stream from the storage module 2 and decode the third encoding code stream. The decoding module 2 can send the decoded data to the desensitization module 2.

[0190] S918: The desensitization module 2 sends relevant data to the encoding module 4 after desensitization detection and desensitization processing on the received image data.

[0191] S919: The encoding module 4 performs secondary encoding on the received relevant data to obtain a fourth encoding code stream. The encoding module 4 can send the fourth encoding code stream to the cloud device through the communication module 2.

[0192] It should be understood that the modules in FIG. 9a are only examples and are not any limitation. In specific implementation, the encoding module 1, the encoding module 2, the encoding module 3 and the encoding module 4 in FIG. 9a can be the same encoding module, and the decoding module 1 and the decoding module 2 can also be the same decoding module. The implementation of each module is not limited in the embodiments of the present application.

[0193] Correspondingly, the cloud device can include a receiving module, a decoding module 3, a decoding module 4, a post-processing module, an ISP module 2, and a downstream task module (not shown in the figure). Optionally, the cloud device can further include a storage module. The receiving module can receive the first encoding code stream from the terminal device and send the first encoding code stream to the decoding module 3. The decoding module 3 can decode the second RAW image from the first encoding code stream by multiplexing the existing encoding and decoding standard, and the second RAW image includes an image in YCoCgDg format, wherein Y is a luminance component, Co and Cg are color components, and Dg is a difference component. The decoding module 3 can send the second RAW image to the post-processing module. The post-processing module can perform post-processing on the second RAW image to obtain the first RAW image. For example, the post-processing step can include at least one of linear transformation processing, color domain inverse transformation processing, and re-stitching processing. The post-processing module can send the first RAW image to the downstream task module to implement machine vision or related task processing based on the RAW domain neural network model in the downstream task module.

[0194] Alternatively, the receiving module can also receive the second encoding code stream from the terminal device and send the second encoding code stream to the decoding module 4. The decoding module 4 can decode the third RAW image from the second encoding code stream by multiplexing the existing encoding and decoding standard, and the third RAW image is the second RAW image after desensitization and includes an image in YCoCgDg format. The decoding module 4 can send the third RAW image to the post-processing module. The post-processing module can perform post-processing on the third RAW image to obtain the first RAW image after desensitization. The post-processing module can send the first RAW image after desensitization to the downstream task module to implement machine vision or related task processing based on the RAW domain neural network model in the downstream task module.

[0195] Optionally, the decoding module 4 can send the decoded first RAW image or the decoded third RAW image to the ISP module 2, and the ISP module 2 can perform other ISP processing on the first RAW image or the third RAW image to obtain an image adapted to the human eye, so as to facilitate the downstream task module to implement the human eye vision related task.

[0196] Alternatively, the receiving module can also receive the third encoding code stream from the terminal device, and send the third encoding code stream to the decoding module 3. The decoding module 3 can decode a fourth image from the third encoding code stream by multiplexing the existing encoding and decoding standard. The fourth image includes a YUV format image, Y is a luminance component, and U and V are color components. The decoding module 3 can send the fourth image to the downstream task module, so as to implement the human eye vision related task in the downstream task module.

[0197] It should be understood that, on the terminal device side, the encoding module 1, the encoding module 2 and the encoding module 3 can be the same encoder of the terminal device, and the decoding module 1 and the decoding module 2 can be the same decoder. The encoder and the decoder can also be the same codec. On the cloud device side, the decoding module 3 and the decoding module 4 can be the same decoder of the cloud device, and the implementation mode of each module is not limited in the embodiments of the application. In specific implementation, the preprocessing module can also call the ISP module 1 to implement the preprocessing process of the RAW domain image. For example, the preprocessing module can call the ISP module 1 to implement the nonlinear transformation processing of the RAW domain image.

[0198] Based on the same concept, the embodiments of the present application also provide an image processing device suitable for the system architecture shown in FIG. 1. Exemplarily, the image processing device can be a vehicle as shown in FIG. 1, or can also be a functional element (such as a plug-in, a component or a chip, etc.) provided in the vehicle, which has the function of implementing the image processing method. In one example, the image processing device can be any device (such as a computing device or a control unit, etc.) with image processing capability in the vehicle, such as a camera, an MDC or a CDC, etc. In another example, the image processing device can be other devices (such as a server or a cloud, etc.) located outside the vehicle, or can also be a functional element provided in other devices with image processing function, which has the function of implementing the image processing method. Optionally, the image processing device can be used to implement the image processing method provided in the above embodiments, or the modules (such as chips) of the image processing device can be used to implement the image processing method provided in the above embodiments, so as to also achieve the beneficial effects possessed by the above embodiments.

[0199] In one example, when the image processing apparatus 1000 is used to implement the image processing method shown in FIG. 3, FIG. 6, FIG. 8, FIG. 9b or FIG. 9c, the image processing apparatus 1000 can include: an acquisition module 1001 configured to acquire a first original RAW image; a preprocessing module 1002 configured to perform preprocessing on the first RAW image to obtain a second RAW image, the second RAW image including an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; an encoding module 1003 configured to encode the second RAW image to obtain a first encoded code stream; a storage module 1004 configured to store the first encoded code stream, or a sending module 1005 configured to send the first encoded code stream to a cloud device. For specific implementation, refer to the method steps implemented in combination with FIG. 3, FIG. 6, FIG. 8, FIG. 9b or FIG. 9c in the foregoing method embodiments, which will not be repeated here.

[0200] In another example, when the image processing apparatus 1100 is used to implement the image processing method shown in FIG. 3, FIG. 6, FIG. 8, FIG. 9b or FIG. 9c, the image processing apparatus 1100 can include: an acquisition module 1101 configured to acquire an encoded code stream; a decoding module 1102 configured to decode the second RAW image from the encoded code stream, the second RAW image including an image in YCoCgDg format, Y being a luminance component, Co and Cg being color components, and Dg being a difference component; and a post-processing module 1103 configured to perform post-processing on the second RAW image to obtain a first RAW image. For specific implementation, refer to the method steps implemented in combination with FIG. 3, FIG. 6, FIG. 8, FIG. 9b or FIG. 9c in the foregoing method embodiments, which will not be repeated here.

[0201] It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division manner can be used. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or can be physically separated, or two or more units can be integrated in one unit. For example, taking the preprocessing module and the encoding module as an example, the preprocessing module and the encoding module can be integrated in one module, or the preprocessing module and the encoding module are the same module. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0202] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art that contributes to the present application or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, etc.) or a processor to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0203] In a simple embodiment, those skilled in the art can conceive that the image processing apparatus in the above embodiments can all adopt the form shown in FIG. 12. The image processing apparatus can be used to implement the technical solutions related to the image processing apparatus in the above method embodiments, and thus can also achieve the beneficial effects possessed by the image processing apparatus in the above method embodiments.

[0204] As shown in FIG. 12, the image processing apparatus 1200 includes a transceiver 1210, a processor 1220, and optionally a memory 1230. The transceiver 1210, the processor 1220, and the memory 1230 are connected to each other. When the image processing apparatus 1200 is used to implement the image processing method provided in the above embodiments, the transceiver 1210 can be used to implement the data transceiving function of the terminal device or the cloud device; the processor 1220 can be used to implement the data processing function of the RAW domain image, or can also be used to implement the data processing function of the RGB domain image or the YUV domain image.

[0205] Optionally, the transceiver 1210, the processor 1220, and the memory 1230 are connected to each other through a bus 1240. The bus 1240 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in FIG. 12, but it does not mean that there is only one bus or only one type of bus.

[0206] The transceiver 1210 is configured to receive and send data. For example, when the transceiver 1210 is deployed in a terminal device, the transceiver 1210 can be configured to realize communication with an image acquisition module or a cloud device. In one example, the transceiver can be a transceiver device integrated with a data transceiving function. In another example, the transceiver can also be composed of a transmitter and a receiver, wherein the transmitter is configured to send data, and the receiver is configured to receive data.

[0207] Optionally, the transceiver 1210 can include a transmitter and / or a receiver. The transmitter is configured to send signals, messages, information or data, etc. The receiver is configured to receive signals, messages, information or data, etc. For example, the transmitter sends signals, messages, information or data, etc. under the control of the processor 1220. The receiver receives signals, messages, information or data, etc. under the control of the processor 1220.

[0208] The functions of the processor 1220 can refer to the method embodiments shown in FIGS. 2-3, which will not be described here. The processor 1220 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP, etc. The processor 1220 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The processor 1220 can be implemented by hardware, and of course can also execute corresponding software to realize the above functions.

[0209] The memory 1230 can include a volatile memory such as a random access memory (RAM), and can also include a non-volatile memory such as at least one disk memory.

[0210] The executable program code is stored in the memory 1230, and the processor 1220 executes the executable program code to respectively implement the functions of the aforementioned image processing system (such as the first module and the second module), thereby implementing the image processing method provided in the embodiments of the present application. That is, the computer program instructions for executing the image processing method are stored on the memory 1230.

[0211] Alternatively, the executable code is stored in the memory 1230, and the processor 1220 executes the executable code to implement the functions of the aforementioned image processing device (such as the preprocessing module or the ISP module), thereby implementing the image processing method provided in the embodiments of the present application. That is, the computer program instructions for executing the image processing method provided in the embodiments of the present application are stored on the memory 1230.

[0212] Based on the same idea, the embodiments of the present application also provide a possible image processing system, which can include one or more of the MDC, the CDC, the industrial computer or the server (or cloud). Optionally, the image processing system can also include a display module for displaying the RAW image. Exemplarily, taking the image processing system including the MDC and the CDC as an example. In one example, the preprocessing module is deployed on the MDC, and the desensitization module is deployed on the CDC. In another example, the first module is deployed on the image acquisition device, and the second module is respectively deployed on the MDC and the CDC. Exemplarily, the number of the MDC or the CDC included in the image processing system can be one or more, and the number of the first module or the second module can be one or more, which are not limited by the embodiments of the present application. The image processing system can be deployed on a vehicle. Accordingly, the embodiments of the present application also provide a vehicle including an image acquisition device and the above-mentioned image processing system. The image acquisition device can be used to acquire the image (such as the RAW image) corresponding to the surrounding environment information of the vehicle.

[0213] Based on the same idea, the embodiments of the present application also provide a computer program product including computer programs or instructions, which, when executed on a computer, cause the computer to execute the image processing method provided in the above embodiments.

[0214] Based on the same idea, the embodiments of the present application also provide a computer readable storage medium, which stores computer programs or instructions, which, when executed by a computer, cause the computer to execute the image processing method provided in the above embodiments.

[0215] The storage medium can be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0216] Based on the same idea, the embodiment of the present application further provides a chip, which is coupled with a memory, and is used for reading a computer program stored in the memory, and implementing the image processing method provided in the above embodiment.

[0217] Based on the same idea, the embodiment of the present application further provides a chip system, which comprises a processor, and is used for supporting a computer device to implement the functions related to the image processing system in the above embodiment. In a possible design, the chip system further comprises a memory, which is used for storing necessary programs and data of the computer device. The chip system can be composed of a chip, or can include the chip and other discrete devices.

[0218] The method provided by the embodiment of the present application can be implemented by software, hardware, firmware or any combination thereof, in whole or in part. When implemented by software, the method can be implemented in the form of a computer program product, in whole or in part. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiment of the present application are generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a high-density digital video disc (digital video disc, DVD)), or a semiconductor medium (such as a solid state drive (solid state drive, SSD)) and the like.

[0219] The steps of a method, process, or algorithm described in connection with the present application can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is tangible and can be accessed by a processor. The processor can read information from the storage medium, and can write information to the storage medium. Alternatively, the storage medium can be integrated into the processor. The processor and the storage medium can reside in an ASIC.

[0220] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0221] These computer program instructions can also be stored in a computer readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the functions specified in the flowchart and / or block diagram block or blocks.

[0222] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An image processing method, characterized by, The method is applied to a terminal device, and comprises the following steps: obtaining a first original RAW image; preprocessing the first RAW image to obtain a second RAW image, wherein the second RAW image comprises an image in YCoCgDg format, Y represents a luminance component, Co and Cg represent color components, and Dg represents a difference component; encoding the second RAW image to obtain a first encoded code stream; storing the first encoded code stream or sending the first encoded code stream to a cloud device.

2. The method of claim 1, wherein, The preprocessing of the first RAW image to obtain the second RAW image comprises the following steps: performing color gamut conversion on the first RAW image to obtain images of four components in the YCoCgDg format; splicing the images of the four components according to a set splicing mode to obtain the second RAW image.

3. The method of claim 2, wherein, The set splicing mode comprises any one of the following: a matrix splicing mode, a horizontal splicing mode, and a vertical splicing mode. The shape of the second RAW image comprises any one of the following: a grid, a horizontal strip, and a vertical strip.

4. The method according to claim 2 or 3, characterized in that, The color gamut conversion on the first RAW image to obtain the images of the four components in the YCoCgDg format comprises the following steps: performing nonlinear conversion on the first RAW image to obtain a converted first RAW image; performing color gamut conversion on the converted first RAW image to obtain the images of the four components in the YCoCgDg format.

5. The method according to any one of claims 2-4, characterized in that, The preprocessing of the first RAW image to obtain the second RAW image further comprises the following steps: performing desensitization detection and desensitization processing based on the image of the luminance component in the four components to obtain a desensitized image of the luminance component; The splicing of the images of the four components according to the set splicing mode to obtain the second RAW image comprises the following steps: splicing the desensitized image of the luminance component, the images of the color components in the four components, and the image of the difference component according to the set splicing mode to obtain the second RAW image.

6. The method according to any one of claims 1-4, characterized in that, The method further comprises the following steps: obtaining the first encoded code stream from a storage medium; decoding the second RAW image from the first encoded code stream; performing desensitization detection and desensitization processing based on the second RAW image to obtain a third RAW image; encoding the third RAW image to obtain a second encoded code stream; sending the second encoded code stream to the cloud device.

7. The method of claim 6, wherein, The desensitization detection and desensitization processing based on the second RAW image to obtain the third RAW image comprises the following steps: performing desensitization detection and desensitization processing based on the image of the luminance component in the second RAW image to obtain a desensitized image of the luminance component; splicing the desensitized image of the luminance component, the images of the color components in the second RAW image, and the image of the difference component according to the set splicing mode to obtain the third RAW image.

8. The method according to any one of claims 1-7, characterized in that, The encoding of the second RAW image to obtain the first encoded code stream comprises the following steps: encoding the second RAW image according to any one of the following encoding standards to obtain the first encoded code stream: H264 coding standard, H265 coding standard, H266 coding standard, JPEG low latency coding standard.

9. The method according to any one of claims 1-8, characterized in that, The terminal device comprises an image acquisition module, and the first original RAW image is acquired. The RAW image corresponding to the current target scene is acquired through the image acquisition module to obtain the first RAW image; or The first RAW image is acquired from a storage medium.

10. The method according to any one of claims 1-9, characterized in that, The terminal device further comprises an image signal processing (ISP) module, and the method further comprises: The first RAW image is processed through the ISP module to obtain a fourth image, wherein the fourth image comprises an image in YUV format The Y is a luminance component, and the U and V are color components; The fourth image is encoded to obtain a third encoding code stream; The third encoding code stream is stored or transmitted to the cloud device.

11. The method according to any one of claims 1-10, characterized in that, The encoder of the terminal device is configured to perform an encoding operation, and the bit width depth of the image supported by the encoder is greater than or equal to 8 bits.

12. An image processing method, characterized by, The method applied to the cloud device comprises: An encoding code stream is acquired. A second original RAW image is decoded from the encoding code stream, wherein the second RAW image comprises an image in YCoCgDg format The Y is a luminance component, the Co and Cg are color components, and the Dg is a difference component.

13. The method of claim 12, wherein, The second RAW image is post-processed to obtain a first RAW image. The second RAW image is post-processed to obtain a first RAW image, comprising: The second RAW image is subjected to color domain inverse transformation to obtain an image in YCoCgDg format; 14. The method of claim 13, wherein, The images of the four components are spliced according to a set splicing mode to obtain the first RAW image. The second RAW image is subjected to color domain inverse transformation to obtain an image in YCoCgDg format, comprising: The second RAW image is subjected to linear transformation to obtain a transformed second RAW image; 15. The method according to any one of claims 12-14, characterized in that, The transformed second RAW image is subjected to color domain inverse transformation to obtain an image in YCoCgDg format. The second original RAW image is decoded from the encoding code stream, comprising: The second original RAW image is decoded from the encoding code stream by using any one of the following encoding standards:

16. The method according to any one of claims 12-15, characterized in that, H264 coding standard, H265 coding standard, H266 coding standard, JPEG low latency coding standard. The encoding code stream is acquired, comprising: The encoding code stream is received from the terminal device; or 17. The method according to any one of claims 12-16, characterized by, The encoding code stream is acquired from a storage medium.

18. An image processing apparatus characterized by comprising: The decoder of the cloud device is configured to perform a decoding operation, and the bit width depth of the image supported by the decoder is greater than or equal to 8 bits. Comprise: An acquisition module is configured to acquire a first original RAW image; A preprocessing module is configured to preprocess the first RAW image to obtain a second RAW image, wherein the second RAW image comprises an image in YCoCgDg format The Y is a luminance component, the Co and Cg are color components, and the Dg is a difference component; An encoding module is configured to encode the second RAW image to obtain a first encoding code stream; and An image signal processing (ISP) module is configured to process the first RAW image to obtain a fourth image, wherein the fourth image comprises an image in YUV format The Y is a luminance component, and the U and V are color components. The storage module is configured to store the first coded stream, or the sending module is configured to send the first coded stream to a cloud device.

19. An image processing apparatus characterized by comprising: The method comprises: The acquisition module is configured to acquire a coded stream. The decoding module is configured to decode a second original RAW image from the coded stream, the second RAW image comprising a YCoCgDg format image, Y being a luminance component, Co and Cg being color components, and Dg being a difference component. The post-processing module is configured to post-process the second RAW image to obtain a first RAW image.

20. A computing device, comprising: The apparatus comprises a processor coupled to a memory: The processor is configured to execute computer programs or instructions stored in the memory, so that the apparatus performs the method of any one of claims 1-11, or so that the apparatus performs the method of any one of claims 12-17.

21. An image processing system, characterized by The apparatus comprises a terminal device and a cloud device, the terminal device being configured to implement the method of any one of claims 1-11, and the cloud device being configured to implement the method of any one of claims 12-17.

22. An image processing system, characterized by The apparatus comprises a transceiver, a memory, and a processor. The transceiver is configured to receive and send data. The memory is configured to store computer program instructions and data. The processor is configured to execute the computer program instructions and data in the memory, so that the image processing system performs the method of any one of claims 1-11, or performs the method of any one of claims 12-17.

23. A vehicle characterized by comprising: The apparatus comprises modules configured to implement the method of any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer programs or instructions, which, when executed by a computer, cause the computer to perform the method of any one of claims 1-11, or perform the method of any one of claims 12-17.

25. A computer program product, characterised in that, The computer program product comprises computer programs or instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1-11, or perform the method of any one of claims 12-17.

26. A chip, characterized by The chip comprises a processor coupled to a memory, configured to execute computer programs or instructions stored in the memory, which, when executed, implement the method of any one of claims 1-11, or implement the method of any one of claims 12-17.

Images