Image processing method, device and electronic equipment

Abstract

The application discloses a processing method and device of an image and electronic equipment. The method comprises the following steps: acquiring a first image to be displayed sent by a second device, wherein the first image is an RGB image; determining a type label of the first image according to a scene type of the first image; performing gamut mapping on the first image to obtain a second image; and sending image data of the second image and the type label to the second device, so that the second device renders the image data of the second image based on the type label and displays the image data after the rendering. The application can balance the display quality and processing efficiency of the image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, and electronic device. Background Technology

[0002] With the development of digital display technology, electronic ink (E-ink) displays are increasingly widely used in art decoration, electronic photo frames, information display, and other scenarios due to their unique display characteristics. Among them, the new generation of color E-ink displays can support six colors: black, white, red, yellow, blue, and green, greatly expanding the application range of E-ink displays.

[0003] E-ink displays have a much smaller color gamut than sRGB, and different scenarios have significantly different requirements for color gamut mapping. Therefore, it is necessary to adaptively optimize the images to be displayed in E-ink based on the image scene type. In related technologies, this optimization is either entirely processed on the device side or entirely on the cloud side. However, device-side computing power is limited and cannot run scene recognition models, so fixed dithering parameters can only be used, making it difficult to guarantee display quality. On the other hand, full cloud processing requires transmitting the entire rendered image data to the device side, resulting in a large data volume, significant waste of bandwidth and cloud computing power, and high response latency. Therefore, the image processing solutions in related technologies cannot simultaneously achieve both image display quality and processing efficiency. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and electronic device, which aims to balance image display quality and processing efficiency.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to a first device, comprising: Obtain a first image to be displayed sent by the second device, wherein the first image is an RGB image; The type label of the first image is determined based on the scene type of the first image; The first image is subjected to color gamut mapping to obtain the second image; The image data of the second image and the type label are sent to the second device so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

[0006] Secondly, embodiments of this application provide an image processing method applied to a second device, comprising: Send a first image to be displayed to a first device, wherein the first image is an RGB image; The device receives the type label of the first image and the image data of the second image sent by the first device, wherein the second image is an image obtained by color gamut mapping of the first image; The image data of the second image is rendered according to the type label, and the rendered image data is displayed.

[0007] Thirdly, embodiments of this application provide an image processing apparatus, comprising: The acquisition module is used to acquire a first image to be displayed sent by the second device, wherein the first image is an RGB image; The determining module is used to determine the type label of the first image based on the scene type of the first image; The mapping module is used to perform color gamut mapping on the first image to obtain the second image; The sending module is used to send the image data of the second image and the type label to the second device, so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

[0008] Fourthly, embodiments of this application provide an electronic device, including: at least one processor; at least one memory for storing at least one program; and when at least one of the programs is executed by at least one of the processors, implementing the image processing method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the image processing method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program or computer instructions, the computer program or computer instructions being stored in a computer-readable storage medium, a processor of a communication device reading the computer program or computer instructions from the computer-readable storage medium, and the processor executing the computer program or computer instructions to cause the communication device to perform the image processing method as described in the first aspect.

[0011] In this embodiment, the first device acquires a first image to be displayed in RGB format sent by the second device, determines the corresponding type label based on the scene type of the first image, performs color gamut mapping on the first image to obtain a processed second image, and finally sends the image data and type label of the second image back to the second device for rendering and displaying based on the type label. This scheme reduces the computational burden on the second device by having the first device handle the scene recognition and color gamut mapping tasks. By delegating the computationally intensive scene recognition and color gamut mapping to the first device, the second device, with its limited computing power, only needs to perform type label-based rendering operations, thereby improving overall processing efficiency while ensuring display quality. Attached Figure Description

[0012] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0013] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application; Figure 2 This is a schematic flowchart of the image processing method provided in the embodiments of this application; Figure 3 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] It should be understood that in the description of the embodiments of this application, the use of terms such as "first" and "second" is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated. "At least one" refers to one or more, and "more" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any group of these items, including any group of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0016] To facilitate understanding of the solutions in the embodiments of this application, some contents involved in the embodiments of this application are described below: With the development of digital display technology, electronic ink (E-ink) displays are increasingly widely used in art decoration, electronic photo frames, information display, and other scenarios due to their unique display characteristics. Among them, the new generation of color E-ink displays can support six colors: black, white, red, yellow, blue, and green, greatly expanding the application range of E-ink displays.

[0017] However, E-ink color display technology still faces many technical challenges in image rendering. Because the color gamut that E-ink displays can display is much smaller than the traditional sRGB color gamut, mapping a color image from the sRGB color space to the E-ink display gamut is a highly non-linear process. Different image types have significantly different requirements for color gamut mapping. For example, portrait scenes require priority to ensure skin tone accuracy, landscape scenes require priority to ensure the smoothness of sky gradients, text scenes require ensuring edge sharpness, and artistic scenes require preserving more texture details.

[0018] Therefore, in related technologies, the image rendering solutions for E-ink photo frame products mainly suffer from the following three types of problems: The first type is the edge-side processing solution. This type of solution completes the entire image processing workflow on the edge microcontroller, including scene recognition, color gamut mapping, and dithering rendering. However, the computing power on the edge is very limited. Running complex dithering algorithms (such as Floyd-Steinberg dithering) to process high-resolution images takes several seconds or even more than ten seconds. Furthermore, it cannot run scene recognition models and can only use fixed dithering parameters. This results in the same rendering strategy being used for images of different scene types, causing problems such as skin tone distortion in portraits, color banding in the sky, and blurred text edges.

[0019] The second type is the fully cloud-based processing solution. This type of solution completes all image rendering on a cloud server and then transmits the rendering results to the device for display via the network. However, transmitting the rendered image data in its entirety results in a huge amount of data, still 1-2MB even after compression. In weak network environments, the transmission time exceeds 10 seconds, and each refresh requires a full transmission, wasting bandwidth and cloud computing power. At the same time, the device on the device needs to wait for the complete data to be received before it can display the image, resulting in high response latency.

[0020] The third category is fixed-parameter rendering schemes. Existing E-ink photo frame products typically use fixed dithering parameters, lacking scene adaptation capabilities and the ability to dynamically adjust rendering strategies based on image content, resulting in limited display quality. For example, when using parameters suitable for portrait scenes to process landscape images, color banding is likely to appear in the sky area; when using parameters suitable for landscape scenes to process text images, the edges of the text are likely to be blurred.

[0021] To address the aforementioned technical problems, this application provides an image processing method, which is applied to a first device. Specifically, as... Figure 1 As shown, the image processing method includes: S101. Obtain the first image to be displayed sent by the second device, wherein the first image is an RGB image.

[0022] In this embodiment, the first device can be a cloud server or an edge computing device with strong computing power, which is used to perform tasks such as scene recognition and color gamut mapping; the second device is an edge E-ink display device, responsible for performing real-time display tasks such as dithering rendering and screen driving. The first device and the second device communicate with each other via a wireless network.

[0023] The first device receives a first image to be displayed from the second device via a wireless communication interface. This first image is an RGB image, meaning each pixel contains image data for three color channels: red, green, and blue. This first image can be received by the second device from the user terminal and forwarded to the first device, or it can be obtained by the second device from local storage and sent to the first device.

[0024] For example, when a user uploads a family photo to a second device via a mobile app, the second device receives the photo via a wireless network and sends it to the first device as the first image, thus the first device obtains the first image to be displayed.

[0025] S102. Determine the type label of the first image based on the scene type of the first image.

[0026] In this embodiment, the first device performs scene analysis on the acquired first image, identifies the scene type to which the image belongs, and determines the corresponding type label based on the identification result. The aforementioned scene type refers to the scene category presented by the content of the first image, which may include at least one of the following: portrait scene, landscape scene, text scene, artistic scene, and mixed scene.

[0027] The scene type can be determined as follows: the first device inputs the first image into a pre-trained image scene classification model, and the classification model outputs the scene type to which the first image belongs. The image scene classification model can be any deep learning model capable of recognizing image scene types, such as lightweight convolutional neural network models like MobileNet-V2.

[0028] In some embodiments, the image scene classification model described above is a MobileNet-V2 model, with a model size of approximately 3.4MB after INT8 quantization. On a GPU-equipped server, the inference time is less than 50ms, and the classification accuracy is greater than 95%. This model can be pre-trained and deployed on a first device. The training process uses an image dataset containing five scene labels: portrait, landscape, text, art, and mixed scenes, as training samples.

[0029] The aforementioned type label is identification information used to characterize the scene type to which the first image belongs. It can be a text string, an enumerated value, or structured data in JSON format. For example, when the type label is a text string, "PORTRAIT" represents a portrait scene, "LANDSCAPE" represents a landscape scene, "TEXT" represents a text scene, "ART" represents an art scene, and "MIXED" represents a mixed scene; when the type label is an enumerated value, such as 0 for a portrait scene, 1 for a landscape scene, 2 for a text scene, 3 for an art scene, and 4 for a mixed scene. In this embodiment, the type label is in the form of a text string to facilitate parsing and processing by the second device.

[0030] For example, after the first device acquires the first image, it inputs it into the MobileNet-V2 scene classification model. The model outputs that the scene type of the image is "portrait". Based on this, the first device determines the type label of the first image as "PORTRAIT".

[0031] S103. Perform color gamut mapping on the first image to obtain the second image.

[0032] In this embodiment, the first device performs color gamut mapping processing on the acquired first image, mapping the first image from the source color gamut sRGB to the color gamut range that the second device can display, resulting in an optimized second image. The purpose of the above color gamut mapping is to map colors in the first image that cannot be accurately displayed on an E-ink display to the closest color representation that the second device can display, while preserving the visual quality and scene features of the image as much as possible.

[0033] For example, after determining that the type label of the first image is "PORTRAIT", the first device performs color gamut mapping processing on the first image for portrait scenes to obtain the second image.

[0034] S104. Send the image data of the second image and the type label to the second device, so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

[0035] In this embodiment, the first device sends the image data and type tag of the second image to the second device via a network communication interface. After receiving the image data and type tag, the second device renders the image data of the second image based on the type tag and displays the rendering result.

[0036] Specifically, the image data of the second image can be compressed and encoded to reduce transmission bandwidth. The aforementioned type tag can be encapsulated together with the image data of the second image in a single data packet or sent separately.

[0037] In this embodiment, the first device acquires a first image to be displayed in RGB format sent by the second device, determines the corresponding type label based on the scene type of the first image, performs color gamut mapping on the first image to obtain a processed second image, and finally sends the image data and type label of the second image back to the second device for rendering and displaying based on the type label. This scheme reduces the computational burden on the second device by having the first device handle the scene recognition and color gamut mapping tasks. By delegating the computationally intensive scene recognition and color gamut mapping to the first device, the second device, with its limited computing power, only needs to perform type label-based rendering operations, thereby improving overall processing efficiency while ensuring display quality.

[0038] As an optional embodiment, the step of performing color gamut mapping on the first image to obtain the second image includes: Convert the first image into a third image in the LAB color space; The third image is compressed in color gamut according to the color gamut compression strategy corresponding to the scene type of the first image to obtain the second image.

[0039] In this embodiment, the first device converts the first image from the RGB color space to the CIELAB color space to obtain a third image in the LAB color space. In the LAB color space, where L... The component represents brightness, a The component represents the color-opposing dimension from green to red, b The component represents the color opposite dimension from blue to yellow. By performing gamut mapping in the LAB color space, the perceptual uniformity of colors can be better maintained, minimizing the color distortion perceived by the human eye during gamut compression.

[0040] The above conversion process from RGB color space to LAB color space can be achieved by first converting RGB values ​​to XYZ color space through the color management module, and then converting them from XYZ color space to LAB color space.

[0041] In this embodiment, the first device can select a corresponding color gamut compression strategy based on the determined scene type of the first image, and perform color gamut compression on the third image in the LAB color space. This compresses color values ​​that exceed the displayable color gamut range of the second device to within the displayable range, thus obtaining the second image. The aforementioned color gamut compression strategy refers to a compression method designed for different scene types, aiming to preserve the most sensitive visual features of the human eye under that scene type. For example, the second device is an E-ink photo frame, which can only display six colors: black, white, red, yellow, blue, and green.

[0042] In this embodiment, the image in the RGB color space is converted to the LAB color space to make the color processing more in line with the characteristics of human eye perception. Then, the corresponding color gamut compression strategy is selected according to the scene type of the image, so as to retain the color accuracy and visual smoothness of the key areas to the greatest extent within the limited display color gamut of the second device, and significantly improve the display quality in different scenes.

[0043] As an optional embodiment, the step of compressing the color gamut of the third image according to the color gamut compression strategy corresponding to the scene type of the first image includes: When the scene type of the first image is a portrait scene, perceptual uniform compression is performed on the skin color region in the third image; When the scene type of the first image is a landscape scene, gradient preservation compression is performed on the sky area in the third image; When the scene type of the first image is an artistic scene, full-gamut uniform compression is performed on the third image; When the scene type of the first image is a text scene, the nearest neighbor mapping is performed on the third image.

[0044] In this embodiment, the first device can select a corresponding color gamut compression strategy based on the scene type of the determined first image, and perform color gamut compression on the third image in the LAB color space to compress color values ​​that exceed the displayable color gamut range of the second device to the displayable range, thereby obtaining the second image.

[0045] Specifically, when the scene type of the first image is a portrait scene, the first device performs perceptual uniform compression on the skin tone region in the third image. The aforementioned skin tone region refers to the pixel region that satisfies the skin tone range in the LAB color space, and can be specifically determined by a skin tone detection algorithm. In the LAB color space, the skin tone region roughly corresponds to a... The amount is 10 to 25, b The region with a component value ranging from 10 to 30.

[0046] The aforementioned perceptual uniform compression refers to the following: for pixels within the skin tone area, a smaller compression ratio is used for color gamut compression to ensure that color changes within the skin tone area transition as smoothly as possible; for pixels outside the skin tone area, a relatively larger compression ratio can be used to prioritize the display quality of the skin tone area.

[0047] In some embodiments, when performing perceptual uniform compression, the first device may also use a skin tone compensation matrix to perform color correction on skin tone regions. This skin tone compensation matrix can be pre-calibrated to optimally map sRGB skin tone regions to the skin tone effect that can be presented by mixing red, yellow, and white among the six colors supported by the second device for display.

[0048] When the scene type of the first image is a landscape scene, the first device performs gradient-preserving compression on the sky region in the third image. The aforementioned sky region refers to the area in the LAB color space... Blueish pixel areas with a component value less than -10.

[0049] The aforementioned gradient-preserving compression refers to the process of preserving the color gradient relationship between adjacent pixels during color gamut compression, especially for pixels in the sky region, to avoid color banding in the sky region due to compression. In practice, an adaptive step-size color gamut compression method can be used, dynamically adjusting the compression ratio based on the color difference between adjacent pixels, ensuring that adjacent pixels with small color differences maintain a small color difference after compression.

[0050] In some embodiments, when performing gradient-preserving compression, the first device may also add a preset parameter of dithering frequency in the blue gradient area, so that when the second device renders the sky area, it can simulate the subtle gradient of the sky with a higher blue grain density.

[0051] When the scene type of the first image is an artistic scene, the first device performs full-gamut uniform compression on the third image. The aforementioned full-gamut uniform compression means applying the same compression standard to all pixel areas in the image, without bias towards any specific color area, to ensure the overall color fidelity of the artwork.

[0052] Art scenes include artworks of various types, such as oil paintings, watercolors, and prints. Because artworks typically contain rich colors and complex textures, they should not be biased in any particular color area, as this could compromise the artist's original creative intent. Full-gamut uniform compression maps the color values ​​of all pixels in the LAB color space evenly to the range that the second device can display, avoiding localized over-compression or biased mapping.

[0053] When the scene type of the first image is a text scene, the first device performs nearest neighbor mapping on the text region in the third image. The aforementioned nearest neighbor mapping refers to directly mapping the color value of each pixel to the closest color among the six colors that the second device can display, without performing complex compression calculations and dithering.

[0054] Since the core requirement for text-based scenarios is to ensure the sharpness and readability of text edges, nearest neighbor mapping avoids the edge blurring problem that dithering algorithms may introduce. For text images that are predominantly black and white, nearest neighbor mapping maps dark text to black, light backgrounds to white, and gray tones to black or white based on a threshold, thus ensuring clear text edges.

[0055] Thus, by selecting the corresponding color gamut compression strategy according to the scene type, this application can perform differentiated processing for different types of image content during the color gamut mapping stage, so that the subsequent rendering of the second device can better preserve the most sensitive visual features of the human eye under each scene type, thereby significantly improving the display quality of the image.

[0056] As an optional embodiment, after performing color gamut mapping on the first image to obtain the second image, the method further includes: If the type label of the first image and the type label of the second image are the same, the first image is divided into multiple image blocks, wherein the second image is the previous frame image of the first image; Each image block of the first image is compared with the corresponding image block in the second image to generate a difference bitmap, wherein the difference bitmap is used to represent the position of the image block that has changed in the first image relative to the second image; The difference bitmap is sent to the second device.

[0057] In this embodiment, to further reduce the amount of data transmitted between the first device and the second device and improve the response speed, an incremental update mechanism is also provided. This incremental mechanism is applicable to scenarios where multiple frames of images are displayed continuously. When two consecutive frames of images have the same scene type, the first device only needs to transmit the changed portion of the image data, rather than the full image data.

[0058] For example, if the second device needs to display multiple images sequentially, the first device processes each image in turn and sends it to the second device. If the type label of the currently processed first image is the same as the type label of the second image sent in the previous frame, the first device performs an incremental update process.

[0059] Specifically, in the incremental update process: the first device divides the first image to be displayed into multiple image blocks according to a preset block division strategy. The block division strategy can be uniform grid division or adaptive block division based on the content of the first image.

[0060] Then, the first device performs a pixel-level comparison between each image block of the current first image and the corresponding image block in the previous frame of the second image to determine whether the two corresponding image blocks are the same or similar, and generates a difference bitmap based on the comparison result. The above comparison can be achieved by calculating the pixel hash values ​​of the two image blocks and comparing whether the hash values ​​are the same; or by calculating the pixel difference between the two image blocks and determining whether the difference is less than a preset threshold.

[0061] The aforementioned difference bitmap is a binary matrix used to represent the positions of image blocks that have changed in the first image relative to the second image. Each element in the difference bitmap corresponds to an image block position. If the image block in the first image at that position is different from the corresponding image block in the second image of the previous frame, the value of the corresponding position in the difference bitmap is set to 1 or a "changed" flag; if there is no change, it is set to 0 or a "no change" flag.

[0062] In some embodiments, the above image block comparison can also be implemented through pixel-level hash caching. Specifically, the second device locally caches the pixel hash value of each image block of the previous frame; when processing the current frame, the first device also calculates the pixel hash value of each image block of the current frame, compares the two hash values, and generates a difference bitmap.

[0063] In this embodiment, after generating the difference bitmap, the first device can send the difference bitmap and type tag to the second device without sending the complete image data. The difference bitmap has a small data size and a short transmission time. After receiving the difference bitmap, the second device can determine the location of the image block for which pixel data needs to be requested based on the changed positions marked in the difference bitmap.

[0064] In this embodiment, by sending only a difference bitmap and type label, rather than the complete second image, to the second device used for image display when the scene type remains unchanged in two consecutive frames, the second device can accurately locate the changed target image blocks based on the difference bitmap and request the pixel data of these image blocks from the cloud. This significantly reduces the amount of data transmitted between the cloud and the device, lowers bandwidth usage and transmission latency, while also reducing the rendering burden on the device and improving the response speed and overall display smoothness in image carousel scenes.

[0065] As an optional embodiment, sending the image data of the second image to the second device includes: Receive a first request sent by the second device, wherein the first request is used to request the first device to send pixel data of a target image block to the second device, wherein the target image block is an image block in the first image that has changed relative to the second image; In response to the first request, pixel data of the target image block is sent to the second device, wherein the image data of the second image is the pixel data of the target image block.

[0066] In this embodiment, after receiving the difference bitmap, the second device can parse the received difference bitmap, identify the positions of all image blocks in the first image that have changed relative to the second image, generate a first request, and send it to the first device. The first request requests the first device to send pixel data of a target image block to the second device, where the target image block is the image block in the first image that has changed relative to the second image.

[0067] The first request mentioned above can take the form of an image block index list, such as {"changed_blocks": [12,15, 23, 45, ...]}, indicating that the second device needs to obtain the pixel data of image blocks numbered 12, 15, 23, 45, etc. It can also take the form of a coordinate list, such as {"changed_regions": [{"x": 0, "y": 0, "w": 16, "h":16}, ...]}.

[0068] In this embodiment, after receiving the first request sent by the second device, the first device extracts the pixel data of the target image blocks from the first image and sends the pixel data of these target image blocks to the second device.

[0069] In this embodiment, through the incremental update mechanism described above, when the scene type of the images to be displayed remains unchanged in two consecutive frames, the actual amount of pixel data transmitted is only the amount of data in the changed areas. Compared to full transmission, this saves a significant amount of bandwidth.

[0070] As an optional embodiment, sending the image data of the second image to the second device includes: If the type label of the first image and the type label of the second image are different, all image data of the second image are sent to the second device.

[0071] In this embodiment, when the first device determines that the type label of the first image to be displayed is different from the type label of the second image in the previous frame, it indicates that the scene category of the two images has changed, for example, switching from portrait to landscape. At this time, due to the differences in color gamut mapping strategies and dithering rendering parameters corresponding to different scenes, the device cannot reuse the rendering result of the previous frame. Therefore, the cloud sends all image data of the second image, i.e., the complete optimized color map, to the second device to ensure that the device can obtain complete rendering materials based on the new scene label and execute the corresponding dithering algorithm, thereby ensuring display quality.

[0072] In this embodiment, when the scene type changes, the device on the edge is able to obtain complete rendering materials that match the new scene by sending complete image data, thereby ensuring display accuracy and image quality when switching between different scenes.

[0073] This application also provides an image processing method. Specifically, such as... Figure 2 As shown, the image processing method includes: S201, Send a first image to be displayed to the first device, wherein the first image is an RGB image; S202, receiving the type label of the first image and the image data of the second image sent by the first device, wherein the second image is an image obtained by color gamut mapping of the first image; S203, render the image data of the second image according to the type label, and display the rendered image data.

[0074] In this embodiment, when the second device needs to display a new first image, it sends the first image to be displayed to the first device via the network, requesting the first device to process the image. Then, the second device receives the processing result returned by the first device, which may include the type label of the first image and the image data of the second image after color gamut mapping.

[0075] In some embodiments, if the type label of the first image is the same as the type label of the previous frame image cached by the second device, the second device may also receive a difference bitmap. In this case, the second device determines the location of the target image block to be requested based on the difference bitmap, sends a first request to the first device, and receives the target image block pixel data returned by the first device. The second device merges the received target image block pixel data with the locally cached previous frame image data to obtain the complete second image.

[0076] In some embodiments, if the type label of the first image is different from the type label of the previous frame image, the second device receives all image data of the second image sent by the first device.

[0077] In this embodiment, the second device can select the corresponding rendering strategy to render the image data of the second image based on the received type label, and then drive the rendering result to display on the screen.

[0078] The above rendering strategy includes the selection of dithering algorithms and parameter sets. Dithering is a technique that simulates continuous tones through spatially distributed pixel arrangement. It is particularly important in E-ink displays because E-ink only supports six-color display, and dithering must be used to simulate intermediate tones. The second device selects different dithering algorithms and parameter sets according to the type label: For example, the rendering strategy for a portrait scene is as follows: the second device selects the Floyd-Steinberg dithering algorithm and uses a portrait parameter set. The diffusion coefficients of this parameter set are [7 / 16, 3 / 16, 5 / 16, 1 / 16]. These four diffusion coefficients correspond to the error allocation weights of the four adjacent pixels on the right, lower left, lower, and lower right, respectively. Meanwhile, in the skin tone region, the diffusion coefficients are halved to [7 / 32, 3 / 32, 5 / 32, 1 / 32] to maintain the smoothness of the skin tone region and avoid excessive diffusion of dithering errors in the skin tone region, which would lead to uneven skin tone. Rendering takes approximately 2.8 seconds.

[0079] For example, the rendering strategy for the landscape scene is as follows: the second device selects the Floyd-Steinberg dithering algorithm and uses a landscape parameter set. This parameter set increases the dithering frequency to twice the standard value in the blue gradient region, resulting in a higher density of blue particles in the sky region, thus simulating a more delicate sky gradient effect. Rendering takes approximately 3.0 seconds.

[0080] For example, the rendering strategy for text scenes is as follows: the second device selects direct six-color nearest neighbor mapping to maintain the sharpness of text edges. This rendering strategy directly maps each pixel to the closest color among the six E-ink colors, without introducing dithering error diffusion and avoiding blurring of text edges. Rendering takes approximately 1.5 seconds.

[0081] For example, the rendering strategy for the art scene is as follows: the second device selects the Atkinson dithering algorithm with a diffusion ratio of 6 / 8 to retain more detail and texture. Compared to the Floyd-Steinberg algorithm, the Atkinson dithering algorithm has a larger error diffusion range, which can better preserve the detail and texture in the artwork. Rendering takes approximately 3.2 seconds.

[0082] In this way, by selecting the corresponding rendering strategy based on the type label through the second device, the edge device can adopt the optimal dithering algorithm and parameters for different types of image content, thereby achieving the best display effect with limited edge computing power.

[0083] In this embodiment, the first device acquires a first image to be displayed in RGB format sent by the second device, determines the corresponding type label based on the scene type of the first image, performs color gamut mapping on the first image to obtain a processed second image, and finally sends the image data and type label of the second image back to the second device for rendering and displaying based on the type label. This scheme reduces the computational burden on the second device by having the first device handle the scene recognition and color gamut mapping tasks. By delegating the computationally intensive scene recognition and color gamut mapping to the first device, the second device, with its limited computing power, only needs to perform type label-based rendering operations, thereby improving overall processing efficiency while ensuring display quality.

[0084] As an optional embodiment, receiving the image data of the second image sent by the first device includes: If the type label of the first image and the type label of the second image are the same, a difference bitmap between the first image and the second image is received, wherein the second image is the previous frame image of the first image, and the difference bitmap is used to represent the position of the image block that has changed in the first image relative to the second image; A first request is generated based on the difference bitmap, wherein the first request is used to request pixel data of a target image block, wherein the target image block is an image block in the first image that has changed relative to the second image; Send the first request to the first device; The first device receives pixel data of the target image block sent by the first device based on the first request, wherein the image data of the second image is the pixel data of the target image block.

[0085] In this embodiment, when the first device determines that the type label of the first image to be displayed is the same as that of the second image in the previous frame, it indicates that the two images belong to the same scene category. At this time, the first device does not need to receive the complete image data, but instead uses an incremental update mechanism. The second device first receives the difference bitmap and type label sent from the cloud. For example, the first device generates a block-level difference bitmap with a granularity of 16×16 pixel blocks, and the data volume is approximately two hundred bytes.

[0086] Based on the received difference bitmap, the second device can determine which image blocks in the first image have changed relative to the second image, and identify these image blocks as target image blocks. Subsequently, the second device generates a first request for pixel data of these target image blocks and sends the first request to the cloud. In response to the first request, the cloud sends the pixel data of the target image blocks to the second device, which receives this pixel data as image data for the second image.

[0087] In this embodiment, when the scene type remains unchanged, the edge device only needs to request and receive the changed image block data, rather than the entire image, thereby significantly reducing the amount of data transmission and bandwidth usage, reducing the rendering computation load of the edge device, and improving the response speed of image switching and the overall display smoothness.

[0088] As an optional embodiment, receiving the image data of the second image sent by the first device includes: If the type label of the first image and the type label of the second image are different, all image data of the second image sent by the first device shall be received.

[0089] In this embodiment, when the first device determines that the type label of the first image to be displayed is different from that of the second image in the previous frame, it indicates that the scene category of the two images has changed, for example, switching from a portrait scene to a landscape scene. At this time, due to the significant differences in the color gamut mapping strategy and dithering rendering parameters corresponding to different scenes, the second device cannot reuse the rendering result of the previous frame image, nor can it transmit only the pixel data of the changed area through the incremental update mechanism. Therefore, the second device directly receives all the image data of the second image sent by the first device, that is, the complete optimized color map. The second device selects the corresponding dithering algorithm and parameter set based on the new type label, renders and displays the received complete image data.

[0090] In this embodiment, when the scene type changes, the second device directly receives the complete image data to ensure that the end side can obtain complete rendering materials that match the new scene, thereby ensuring the display accuracy and image quality when switching between different scenes.

[0091] As an optional embodiment, the method further includes: If the type label of the first image is not received, the first image is rendered using the rendering parameters of the second image.

[0092] In this embodiment, considering the possibility of network interruptions or other abnormal situations in the network connection between the second device and the first device, an offline degradation strategy is also provided to ensure that the second device can still display images normally when the network is unavailable.

[0093] For example, an offline degradation strategy can render using the most recent set of scene parameters cached.

[0094] In some embodiments, the second device caches the most recently successfully received type tag and its corresponding rendering parameter set in its local memory. When the second device is unable to receive the type tag and other data sent by the first device due to network disconnection or other reasons, and the second device needs to display a new first image, it directly uses the cached rendering parameter set corresponding to the type tag to render the image.

[0095] For example, the offline degradation strategy can also enable local simplified scene detection.

[0096] In some embodiments, the second device may also enable a local simplified scene detection function to classify scenes based on the histogram features of the image, so as to automatically determine appropriate rendering parameters when the network connection is interrupted. The aforementioned local simplified scene detection can be a rule-based classification method, such as: calculating the color distribution entropy of the image, and if the entropy is greater than 4.5, it is determined to be an artistic scene; detecting the proportion of skin color pixels in the image, and if the proportion of skin color pixels is greater than 15%, it is determined to be a portrait scene; detecting the proportion of blue pixels in the image, and if the proportion of blue pixels is greater than 20% and is mainly distributed in the upper half, it is determined to be a landscape scene; if the proportion of black and white pixels in the image is greater than 90%, it is determined to be a text scene.

[0097] In some embodiments, if the second device does not receive the type label of the first image sent by the first device, the second device can render the image using the cached most recent scene parameter set, or use local simplified scene detection to determine the rendering parameters, thereby ensuring basic display functionality.

[0098] In this embodiment, by employing an offline degradation strategy, the second device can still display images normally even if the network connection is lost, effectively improving the reliability of the system.

[0099] It should be noted that the various embodiments described in this application can be combined with each other or implemented independently without conflict, and this application does not limit this. It should also be noted that, without conflict, the embodiments and features described in this application can be combined with each other.

[0100] Based on the image processing method provided in the above embodiments, this application also provides specific implementations of an image processing apparatus. Please refer to the following embodiments.

[0101] For ease of understanding, the following specific embodiments are provided as examples: Example 1: This application example describes a specific implementation of applying the image processing method of this application to a home E-ink color photo frame carousel scenario.

[0102] Hardware Configuration: The second device uses a microcontroller module (such as ESP32-S3, dual-core 240MHz, 8MB PSRAM, 16MB Flash) integrating WiFi (Wireless Fidelity) and Bluetooth, paired with a 13.3-inch E-ink Spectra 6 display (1600x1200 resolution, supporting six colors: black / white / red / yellow / blue / green), driven by a T2000 TCON timing controller, powered by USB-C 5V / 2A, and connected to a home router (802.11n / ac) via WiFi. The first device runs a MobileNet-V2 scene classification model (INT8 quantization, model size 3.4MB, inference time 35ms) and a CIELAB color gamut mapping module. The first device is a cloud-based device, while the second device is an E-ink photo frame on the device side.

[0103] The workflow is as follows: Step A1: The user uploads a family photo in RGB format with a resolution of 1600x1200 as the first image to the second device via a mobile app.

[0104] Step A2: The second device sends the photo as the first image to the first device via WiFi.

[0105] Step A3: After receiving the first image, the first device inputs it into the MobileNet-V2 scene classification model for scene recognition. The model classifies the photo as "portrait" with a classification confidence level of 98%. Based on this, the first device determines the type label as "PORTRAIT".

[0106] Step A4: The first device converts the first image from the RGB color space to the LAB color space to obtain the third image. Then, it performs color gamut compression on the third image according to the color gamut compression strategy corresponding to the portrait scene: it detects skin color areas, performs perceptual uniform compression on skin color areas, and performs standard color gamut compression on non-skin color areas. The compressed image is then converted back to the RGB color space to obtain the second image.

[0107] Step A5: The first device compresses the second image into JPEG format, encapsulates the compressed image data and the type tag "PORTRAIT" in a JSON message body, and sends it to the second device. The message format is: {"scene_type": "PORTRAIT", "image_data":<JPEG binary data>}.

[0108] Step A6: The second device receives the type tag and the image data, parses out the "PORTRAIT" tag, and selects the portrait parameter set: Floyd-Steinberg dithering algorithm, standard diffusion coefficients [7 / 16, 3 / 16, 5 / 16, 1 / 16], and the diffusion coefficients for the skin color region are halved to [7 / 32, 3 / 32, 5 / 32, 1 / 32]. The second device performs dithering rendering on the second image, which takes about 2.8 seconds.

[0109] Step A7: The second device drives the T2000 TCON through the SPI interface to perform a full refresh, which takes about 1.5 seconds, and finally displays the rendered family photo on the E-ink display screen. The skin color of the person in the photo is natural, the background is smooth, and the overall display effect is good.

[0110] Step A8: After 30 minutes, the carousel switches to the next family photo. After processing the new photo, the first device finds that its scene type is also "PORTRAIT", which is the same as the type tag of the previous frame, so the incremental update mechanism is enabled: a 16x16 pixel block-level comparison is performed between the new image and the previous frame image to generate a difference bitmap. Assuming that the difference bitmap this time shows that about 12% of the pixel blocks have changed, the first device sends the difference bitmap and the type tag to the second device first; the second device requests the data of the changed region according to the difference bitmap; the first device sends the pixel data of the target image block. The second device merges the received changed region data with the previous frame image cached locally, and re-performs dithering rendering and display refresh.

[0111] Through the incremental update mechanism, the total data volume of a single refresh is reduced from 850KB to about 110KB, saving 87% of bandwidth, and the total time consumption is reduced from 4.5 seconds to about 3.8 seconds.

[0112] Embodiment 2: This application example describes the specific implementation of applying the image processing method of the present application to the E-ink display scenario of an art gallery.

[0113] The hardware configuration is the same as that of the first embodiment, and the displayed content is replaced with art paintings, where the art paintings include oil paintings, watercolor paintings, prints and the like.

[0114] The working process is as follows: Step B1: The gallery administrator pushes an RGB format oil painting with a resolution of 1600x1200 to the second device through the management platform.

[0115] Step B2: The second device sends the oil painting as the first image to the first device.

[0116] Step B3: After receiving the first image, the first device inputs it into the MobileNet-V2 scene classification model. The model classifies the painting as an "art" scene with a classification confidence level of 96%. Based on this, the first device determines the type label as "ART".

[0117] Step B4: The first device converts the first image to the LAB color space and performs color gamut compression on the third image according to the color gamut compression strategy corresponding to the art scene—uniform compression across the entire color gamut. Uniform compression applies the same compression standard to all pixel areas in the image, without biasing towards any specific color area, ensuring that the rich colors in the oil painting are presented in a balanced manner.

[0118] Step B5: The first device sends the compressed second image and the type label "ART" to the second device.

[0119] Step B6: The second device receives the type label "ART" and selects the Atkinson dithering algorithm to render the second image. The Atkinson dithering algorithm has a larger error spread range than the Floyd-Steinberg algorithm, which can better preserve the brushstroke texture and detail levels in the oil painting. Rendering takes approximately 3.2 seconds.

[0120] Step B7: The second device performs a full refresh via SPI to drive the T2000 TCON, which takes approximately 1.5 seconds, ultimately displaying the rendered oil painting on the E-ink display. The painting features rich colors, clear textures, and natural transitions between layers.

[0121] Step B8: Offline Degradation Test. After disconnecting the second device's WiFi connection, the second device uses the locally cached "ART" parameter set and performs local simplified scene detection based on histogram features.

[0122] This application also provides an image processing apparatus, such as... Figure 3 As shown, the image processing apparatus 300 includes: The acquisition module 301 is used to acquire a first image to be displayed sent by the second device, wherein the first image is an RGB image; The determining module 302 is used to determine the type label of the first image based on the scene type of the first image; Mapping module 303 is used to perform color gamut mapping on the first image to obtain a second image; The sending module 304 is used to send the image data of the second image and the type label to the second device, so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

[0123] In this embodiment, the first device acquires a first image to be displayed in RGB format sent by the second device, determines the corresponding type label based on the scene type of the first image, performs color gamut mapping on the first image to obtain a processed second image, and finally sends the image data and type label of the second image back to the second device for rendering and displaying based on the type label. This scheme reduces the computational burden on the second device by having the first device handle the scene recognition and color gamut mapping tasks. By delegating the computationally intensive scene recognition and color gamut mapping to the first device, the second device, with its limited computing power, only needs to perform type label-based rendering operations, thereby improving overall processing efficiency while ensuring display quality.

[0124] The image processing apparatus provided in this application embodiment can achieve Figure 1 The various processes implemented by the interference signal identification device in the method embodiment will not be described again here to avoid repetition.

[0125] This application also provides an electronic device, such as... Figure 4 As shown, the electronic device 1400 includes: One or more processors 1410; The memory 1420 stores one or more programs that, when executed by one or more processors 1410, cause the one or more processors 1410 to implement the image processing method described in any of the above embodiments.

[0126] Memory 1420, as a non-transitory network system, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 1420 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1420 may optionally include remotely located memories 1420 relative to processor 1410, which can be connected to processor 1410 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] The memory 1420 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1420 and is called and executed by the processor 1410.

[0128] The processor 1410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0129] In some embodiments, the electronic device further includes: Input / output interfaces are used to implement information input and output; The communication interface is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus transmits information between various components of the device (e.g., processor 1410, memory 1420, input / output interfaces, and communication interfaces); The processor 1410, memory 1420, input / output interface, and communication interface can communicate with each other within the device via a bus.

[0130] One embodiment of this application also provides a computer-readable storage medium storing computer-executable instructions for performing an image processing method as provided in any embodiment of this application.

[0131] An embodiment of this application also provides a computer program product, including a computer program or computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform an image processing method as provided in any embodiment of this application.

[0132] The system architecture and application scenarios described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will understand that as system architectures evolve and new application scenarios emerge, the technical solutions provided in this application are also applicable to similar technical problems.

[0133] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0134] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0135] The above description, with reference to the accompanying drawings, illustrates some embodiments of this application, but does not limit the scope of this application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of this application.

Claims

1. An image processing method, the method being applied to a first device, comprising: Obtain a first image to be displayed sent by the second device, wherein the first image is an RGB image; The type label of the first image is determined based on the scene type of the first image; The first image is subjected to color gamut mapping to obtain the second image; The image data of the second image and the type label are sent to the second device so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

2. The image processing method according to claim 1, characterized in that, The step of performing color gamut mapping on the first image to obtain the second image includes: Convert the first image into a third image in the LAB color space; The third image is compressed in color gamut according to the color gamut compression strategy corresponding to the scene type of the first image to obtain the second image.

3. The image processing method according to claim 2, characterized in that, The step of compressing the color gamut of the third image according to the color gamut compression strategy corresponding to the scene type of the first image includes: When the scene type of the first image is a portrait scene, perceptual uniform compression is performed on the skin color region in the third image; When the scene type of the first image is a landscape scene, gradient preservation compression is performed on the sky area in the third image; When the scene type of the first image is an artistic scene, full-gamut uniform compression is performed on the third image; When the scene type of the first image is a text scene, the nearest neighbor mapping is performed on the third image.

4. The image processing method according to claim 1, characterized in that, After performing color gamut mapping on the first image to obtain the second image, the method further includes: If the type label of the first image and the type label of the second image are the same, the first image is divided into multiple image blocks, wherein the second image is the previous frame image of the first image; Each image block of the first image is compared with the corresponding image block in the second image to generate a difference bitmap, wherein the difference bitmap is used to represent the position of the image block that has changed in the first image relative to the second image; The difference bitmap is sent to the second device.

5. The image processing method according to claim 4, characterized in that, The step of sending the image data of the second image to the second device includes: Receive a first request sent by the second device, wherein the first request is used to request the first device to send pixel data of a target image block to the second device, wherein the target image block is an image block in which the first image changes relative to the second image; In response to the first request, pixel data of the target image block is sent to the second device, wherein the image data of the second image is the pixel data of the target image block.

6. The image processing method according to claim 1, characterized in that, The step of sending the image data of the second image to the second device includes: If the type label of the first image and the type label of the second image are different, all image data of the second image are sent to the second device.

7. An image processing method, the method being applied to a second device, comprising: Send a first image to be displayed to a first device, wherein the first image is an RGB image; The device receives the type label of the first image and the image data of the second image sent by the first device, wherein the second image is an image obtained by color gamut mapping of the first image; The image data of the second image is rendered according to the type label, and the rendered image data is displayed.

8. The image processing method according to claim 7, characterized in that, The image data received from the second image sent by the first device includes: If the type label of the first image and the type label of the second image are the same, a difference bitmap between the first image and the second image is received, wherein the second image is the previous frame image of the first image, and the difference bitmap is used to represent the position of the image block that has changed in the first image relative to the second image; A first request is generated based on the difference bitmap, wherein the first request is used to request pixel data of a target image block, wherein the target image block is an image block in the first image that has changed relative to the second image; Send the first request to the first device; Receive pixel data of the target image block sent by the first device based on the first request, wherein the image data of the second image is the pixel data of the target image block.

9. The image processing method according to claim 7, characterized in that, The image data received from the second image sent by the first device includes: If the type label of the first image and the type label of the second image are different, all image data of the second image sent by the first device shall be received.

10. The image processing method according to claim 9, characterized in that, The method further includes: If the type label of the first image is not received, the first image is rendered using the rendering parameters of the second image.

11. An image processing apparatus, comprising: The acquisition module is used to acquire a first image to be displayed sent by the second device, wherein the first image is an RGB image; The determining module is used to determine the type label of the first image based on the scene type of the first image; The mapping module is used to perform color gamut mapping on the first image to obtain the second image; The sending module is used to send the image data of the second image and the type label to the second device, so that the second device renders the image data of the second image based on the type label and displays the rendered image data.

12. An electronic device, comprising: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the image processing method as described in any one of claims 1 to 10.

Images