Image processing method and display device

Abstract

The application relates to the technical field of image processing, in particular to an image processing method and a display device, which are used for improving image processing efficiency, and the method is as follows: a display device obtains to-be-processed cache data corresponding to a to-be-processed image through rendering, then caches the to-be-processed cache data according to a mapping relationship between pixels and storage positions, and then, in response to an image processing operation, acquires each texture feature data corresponding to a pixel according to the mapping relationship, and obtains a target image according to each texture feature data. In this way, the texture feature data corresponding to the to-be-processed image is acquired through the mapping relationship, and subsequent processing can be directly performed on the texture feature data instead of directly processing the pixels, so that the image processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method and display device. Background Technology

[0002] With the continuous development of computer technology, it is often necessary to process images, such as cropping, compression, and stretching. In related technologies, image processing typically involves processing pixels within the image. However, pixel-level processing consumes a significant amount of CPU time, resulting in low image processing efficiency. Summary of the Invention

[0003] This application provides an image processing method and a display device to solve the technical problem in the prior art that high CPU usage and low image processing efficiency are caused by processing pixels.

[0004] The specific technical solutions provided in this application are as follows:

[0005] In a first aspect, a display device includes:

[0006] A monitor is used to display the user interface.

[0007] The controller is configured as follows:

[0008] The acquired image to be processed is rendered to obtain the texture information of the image to be processed. Based on the texture information, the cache data to be processed corresponding to the image to be processed is obtained. Based on the mapping relationship between pixels and storage locations, the cache data to be processed is cached. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel.

[0009] In response to the image processing operation, the storage location of each texture feature data in the cache data to be processed is determined according to the mapping relationship, and the texture feature data is obtained according to the determined storage location, and the target image is obtained according to the texture feature data.

[0010] In this embodiment, during the rendering of the image to be processed, cached data is cached based on the mapping relationship between pixels and storage locations. Then, in response to image processing operations, the texture feature data corresponding to each pixel in the image to be processed can be quickly obtained based on the above mapping relationship. In this way, the texture feature data is processed directly instead of the pixels, which reduces the amount of computation, thereby reducing the CPU usage and improving the image processing efficiency.

[0011] Secondly, an image processing method includes:

[0012] The acquired image to be processed is rendered to obtain the texture information of the image to be processed. Based on the texture information, the cache data to be processed corresponding to the image to be processed is obtained. Based on the mapping relationship between pixels and storage locations, the cache data to be processed is cached. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel.

[0013] In response to the image processing operation, the storage location of each texture feature data in the cache data to be processed is determined according to the mapping relationship, and the texture feature data is obtained according to the determined storage location, and the target image is obtained according to the texture feature data.

[0014] In some embodiments, obtaining the target image based on the respective texture feature data includes:

[0015] The texture feature data are input into the joint detection model to obtain the joints of the target object contained in the image to be processed.

[0016] The various joints are labeled onto the image to be processed to obtain the target image.

[0017] In some embodiments, the image format corresponding to the cached data to be processed is a camera image format; then, inputting the various texture feature data into the joint detection model to obtain the joints of the target object contained in the image to be processed includes:

[0018] Based on the data conversion relationship between the camera image format and the display device image format, each texture feature data is converted into target cache data. The image format corresponding to the target cache data is the display device image format, and the camera image format is different from the display device image format.

[0019] The target cache data is input into the joint detection model to obtain each joint of the target object contained in the image to be processed.

[0020] Thirdly, an image processing apparatus includes:

[0021] A caching unit is used to render the acquired image to be processed to obtain the texture information of the image to be processed, and to obtain the cache data to be processed corresponding to the image to be processed based on the texture information, and to cache the cache data to be processed based on the mapping relationship between pixels and storage locations. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel.

[0022] The processing unit is configured to, in response to an image processing operation, determine the storage location of each texture feature data in the cache data to be processed according to the mapping relationship, acquire each texture feature data based on the determined storage location, and obtain the target image based on the each texture feature data.

[0023] Fourthly, a storage medium, when instructions in the storage medium are executed by a processor, enables the processor to perform the image processing method as described in any of the preceding claims. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application;

[0026] Figure 2 This is a hardware configuration block diagram of the display device 200 provided in the embodiments of this application;

[0027] Figure 3 This is a hardware configuration block diagram of the control device 100 provided in the embodiments of this application;

[0028] Figure 4 This is a schematic diagram of the software configuration of the display device 200 provided in the embodiments of this application;

[0029] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 according to an embodiment of this application;

[0030] Figure 6 This is a flowchart illustrating the image processing method provided in the embodiments of this application;

[0031] Figure 7This is a schematic diagram illustrating the storage method of a single pixel provided in the embodiments of this application;

[0032] Figure 8 This is a schematic diagram illustrating the storage method of each pixel provided in the embodiments of this application;

[0033] Figure 9 This is a schematic diagram illustrating the data storage method provided in the embodiments of this application;

[0034] Figure 10A This is a schematic diagram of an image to be processed provided in an embodiment of this application;

[0035] Figure 10B This is a schematic diagram of a target image provided in an embodiment of this application;

[0036] Figure 11 This is a schematic diagram of the target cropping area provided in the embodiments of this application;

[0037] Figure 12 This is a schematic diagram of the nearest neighbor algorithm provided in the embodiments of this application;

[0038] Figure 13A This is a schematic diagram of another image to be processed provided in the embodiments of this application;

[0039] Figure 13B This is a schematic diagram of another target image provided in an embodiment of this application;

[0040] Figure 14 This is a flowchart illustrating another image processing method provided in the embodiments of this application;

[0041] Figure 15 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0042] To address the low image processing efficiency caused by pixel-level processing in existing technologies, this application provides an image processing solution to improve efficiency. The solution involves: after rendering the acquired image to be processed, obtaining corresponding cached data for the image to be processed; caching this cached data according to the mapping relationship between pixels and storage locations; then, during the image processing stage, determining the storage location of each texture feature data within the cached data based on the aforementioned mapping relationship; and finally, acquiring each texture feature data based on the determined storage locations, and ultimately obtaining the target image based on the texture feature data.

[0043] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0044] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0045] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete implementation on its own. It should be noted that the brief descriptions of terminology in this application are merely for the convenience of understanding the embodiments described below, and are not intended to limit the implementation of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0046] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1 As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0047] In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to achieve connection and communication via network communication protocols, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronous display. The display device 200 also communicates with the server 400 via various communication methods. The display device 200 can communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactive features to the display device 200. The display device 200 can be a liquid crystal display, an OLED display, or a projection display device. In addition to providing broadcast television reception functions, the display device 200 can also be equipped with a smart network television function that provides computer support.

[0048] Figure 2An exemplary block diagram of the configuration of the control device 100 according to an exemplary embodiment is shown. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input commands and convert them into commands that the display device 200 can recognize and respond to, acting as an intermediary for interaction between the user and the display device 200. The communication interface 130 is used for external communication and includes at least one of a Wi-Fi chip, a Bluetooth module, NFC, or a replacement module. The user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, buttons, or a replacement module.

[0049] Figure 3 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 3 The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 can be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and can also be a projection device and a projection screen. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 can be located in different separate devices; that is, the tuner / demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0050] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200. The user can input user commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, the user can input user commands by inputting specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0051] In some embodiments, a "user interface" is the medium through which an application or operating system interacts and exchanges information with a user, enabling the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include at least one of the visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0052] Figure 4 This is a schematic diagram of the software configuration in a display device 200 according to one or more embodiments of this application, such as... Figure 4 As shown, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the Android runtime and system library layer (referred to as the "System Runtime Layer"), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0053] Figure 5 This is a schematic diagram showing the icon control interface of an application in a display device 200 according to one or more embodiments of this application, such as... Figure 5 As shown, the application layer contains at least one application whose corresponding icon control can be displayed on the screen, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV applications can provide live television from different signal sources. Video-on-demand applications can provide video from different storage sources. Unlike live TV applications, video-on-demand provides video display from certain storage sources. Media center applications can provide applications for playing various multimedia content. The application center can provide storage for various applications.

[0054] See Figure 6 As shown, it is a flowchart of a graphics processing method provided in an embodiment of this application. The method can be applied to the above-mentioned display device 200. Specifically, the process includes the following steps.

[0055] S601. The display device renders the acquired image to be processed to obtain the texture information of the image to be processed, and obtains the processing cache data corresponding to the image to be processed based on the texture information, and caches the processing cache data according to the mapping relationship between pixels and storage locations. The image to be processed contains each pixel, and the processing cache data stores the texture feature data corresponding to each pixel.

[0056] Specifically, the display device can render the acquired image to be processed by calling the GLsurfaceview method. The GLsurfaceview method is responsible for rendering using the Open Graphics Library (OpenGL), and the texture information of the image to be processed can be obtained through the GLsurfaceview method. Correspondingly, the display device can retrieve cached data to be processed from the cache using the OpenGL glreadpixel method.

[0057] Texture refers to the pattern created by the variation of colors in space. Texture information can include, but is not limited to, grayscale and color information.

[0058] The texture feature data stored in the cache data to be processed can be values ​​of R, G, B, and A, where R, G, B, and A represent Red, Green, Blue, and Alpha, respectively. Alpha is generally used as an opacity parameter.

[0059] In this embodiment, the cached data to be processed is stored in bytebuffer format. Taking a red-green-blue (RBG) format image to be processed as an example, each pixel in the image to be processed contains four values: R, G, B, and A. Each value is represented by 8 bits in the cached data to be processed, that is, each pixel is represented by 4 bytes in the cached data to be processed.

[0060] For example, see Figure 7 As shown, a pixel in the image to be processed is red. The R, G, B, and A values ​​of this pixel are 255, 0, 0, and 0, respectively. These values ​​can be stored in the processing buffer data using 8 bits each.

[0061] In this application, the mapping relationship between pixels in the image to be processed and their storage locations is described in [reference needed]. Figure 8 As shown. For example, in the image to be processed, there are 1920 pixels in each row and 1080 pixels in each column. Therefore, the image to be processed has a total of 1920*1080 pixels. Taking pixel 1 as an example, the texture feature data of pixel 1 is stored in the storage location shown by the dashed line in the figure. In this application, the image to be processed can also be called a bitmap.

[0062] Since the stored data is itself a set of data and has no practical meaning, in this embodiment of the application, the cached data to be processed stored in bytebuffer format can be converted into cached data to be processed stored in array format. Since the array itself is stored contiguously, the data in the cached data to be processed stored in array format has continuity.

[0063] For information on how arrays are stored in memory, please refer to [link / reference]. Figure 9 As shown, an array is a reference data type. The actual array elements (element 1, element 2, ..., element 5) are stored in the heap memory, while the array reference variable (i.e., array variable p) is a reference type variable stored in the stack memory.

[0064] Since the data is stored in a contiguous array, which looks like a bitmap but is actually a series of data, the storage location of a pixel's texture feature data can be located based on the row and column correspondence, that is, the mapping relationship between pixels and storage locations.

[0065] It should be noted that the acquired image to be processed may be in landscape or portrait mode. Landscape mode refers to an image whose width is greater than its height, while portrait mode refers to an image whose width is no greater than its height. In this embodiment, to further improve processing efficiency, after acquiring the image to be processed, if the image is not in the target state, it is rotated to the target state, and then subsequent processing is performed based on the rotated image. The target state can be either landscape or portrait; this application does not impose any limitation on this. The following description will only use landscape mode as the target state.

[0066] S602. The display device responds to the image processing operation by determining the storage location of each texture feature data in the cache data to be processed according to the mapping relationship, and obtaining each texture feature data according to the determined storage location, and obtaining the target image according to each texture feature data.

[0067] During the process of acquiring each texture feature data, since the data caching is performed based on the mapping relationship, in S602, the display device can determine the storage location of each texture feature data in the cache data to be processed according to the mapping relationship, and acquire each texture feature data based on the determined storage location.

[0068] When a display device obtains a target image based on various texture feature data, it may perform at least one of the following operations:

[0069] Operation 1: The display device determines the texture feature data to be cropped corresponding to the target pixels contained in the target cropping area from various texture feature data, and obtains the target image based on the texture feature data to be cropped.

[0070] For example, see Figure 10A and Figure 10B As shown, Figure 10A The image to be processed contains a target cropping region, which contains a human figure image. The display device can determine the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region from various texture feature data, and obtain the cropping texture feature data based on the cropping texture feature data. Figure 10B The target image shown.

[0071] It should be noted that, in the embodiments of this application, the target cropping area may be, but is not limited to, rectangular, circular, square, or other graphic areas. For example, see [reference needed]. Figure 11 As shown, the target cropping region is a rectangular region in the image to be processed. The two vertices of the target cropping region are (x1, y1) and (x2, y2). In the following text, the rectangular region can be represented as (x1, y1, x2, y2).

[0072] Specifically, when a display device determines the texture feature data to be cropped corresponding to the target pixel in the target cropping region from various texture feature data for the target cropping region contained in the image to be processed, it can determine the cropping start position and then obtain the texture feature data to be cropped corresponding to the target pixel from the cropping start position.

[0073] The starting position for cutting can be calculated using the following formula (1):

[0074] offset=y*stride+x*4 formula (1)

[0075] Where offset represents the starting position of the crop, (x, y) represents the coordinates of the pixel corresponding to the starting position of the crop in the image to be processed, and stride represents the stride of a row of images. Since 4 bytes are used to store one pixel, the value of stride is width*4, where width represents the width of the image to be processed.

[0076] Taking a rectangular region (x1, y1, x2, y2) as an example, assume that the coordinates of the pixel corresponding to the cropping start position in the image to be processed are (x1, y1), and the cropping start position offset = y1*1920*4+x1*4. The display device starts from the cropping start position and obtains the texture feature data to be cropped corresponding to the target pixel until it ends at y2*1920*4+x2*4, that is, the storage location of the texture feature data of the pixel at (x2, y2) ends.

[0077] Compared to existing technologies that directly crop the bitmap, this application reduces processing time by two-thirds and improves processing efficiency by performing image processing based on the cached data to be processed.

[0078] Operation 2: The display device determines the compression ratio based on the size of the image to be processed, and obtains the target image based on the compression ratio and various texture feature data.

[0079] The compression ratio can include the width compression ratio and the height compression ratio. The display device can determine the width compression ratio and the height compression ratio according to the size of the image to be processed and the size of the target image, respectively.

[0080] In some possible embodiments, the display device obtains the target image by using a preset interpolation algorithm based on the compression ratio and various texture feature data. The preset interpolation algorithm may be, but is not limited to, the nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and other interpolation algorithms.

[0081] Among them, the nearest neighbor method has the fastest calculation speed, but the effect is relatively poor; the bilinear interpolation algorithm calculates 1 point in the compressed image based on 4 (2*2) points in the original image, and the effect is slightly worse than bicubic interpolation, but the speed is faster than bicubic interpolation; the bicubic interpolation algorithm calculates 1 point in the new image based on 16 (4*4) points in the original image, and the effect is better, but the calculation speed is slower.

[0082] In practical applications, the appropriate interpolation algorithm can be selected for calculation based on the application scenario. Taking the human joint recognition scenario as an example, since the requirements for image quality are not high, the nearest neighbor method can be selected for calculation.

[0083] See below Figure 12 As shown, the nearest neighbor method is introduced using an image as an example.

[0084] Assuming the width and height of the compressed image are DW and DH respectively, and the width and height of the original image are SW and SH respectively, for a point (Sx, Sy) in the original image, the corresponding coordinates (Dx, Dy) in the compressed image can be calculated. Dx can be calculated according to formula (2), and Dy can be calculated according to formula (3):

[0085] Dx=Sx*DW / SW Formula (2)

[0086] Dy=Sy*DH / SH Formula (3)

[0087] In other embodiments, the display device may also obtain the target image in the following manner:

[0088] A1. The display device determines the data acquisition interval based on the compression ratio.

[0089] Specifically, the display device can use the width compression ratio in the compression ratio as the data acquisition interval. For example, the display device can use the SW / DW value as the data acquisition interval.

[0090] A2. The display device selects the texture feature data to be compressed from each texture feature data according to the data acquisition interval.

[0091] Specifically, the display device can determine the starting position based on the compression ratio. The starting position represents the position of the center point of the image to be processed in the buffer data to be processed. Then, starting from the starting position, the texture feature data to be compressed is selected from each texture feature data according to the data acquisition interval.

[0092] It should be noted that, in this embodiment of the application, in order to improve compression efficiency, the display device can determine the compression area based on the starting position, and then, for each texture feature data contained in the compression area, starting from the starting position, select the texture feature data to be compressed from each texture feature data contained in the compression area according to the data acquisition interval.

[0093] For example, `dispRatio` represents the aspect ratio of the screen, and `imgRatio` represents the aspect ratio of the image to be processed. `dispRatio` can be calculated using formula (4), and `imgRatio` can be calculated using formula (5). In formula (4), `mSurfaceWidth` represents the width of the screen, and `mSurfaceHeight` represents the height of the screen. In formula (5), `mImageWidth` represents the width of the image to be processed, and `mImageHeight` represents the height of the image to be processed. It should be noted that in this embodiment, "screen" refers to the surface window in the display device.

[0094] dispRatio=mSurfaceWidth / mSurfaceHeight formula (4)

[0095] imgRatio=mImageWidth / mImageHeight formula (5)

[0096] The upper left corner coordinates of the compressed area are (xt, yt), the width of the compressed area is wt, and the height of the compressed area is ht. If the aspect ratio of the screen is not less than the aspect ratio of the image to be processed, then xt is represented by formula (6), yt is 0, wt is represented by formula (7), and ht is represented by formula (8).

[0097] xt=(mImageWidth-mImageHeight*mSurfaceWidth / mSurfaceHeight) / 2Formula (6)

[0098] wt=mImageWidth-xt formula (7)

[0099] ht=mImageHeight formula (8)

[0100] If the aspect ratio of the screen is less than the aspect ratio of the image to be processed, then the value of xt is 0, yt is represented by formula (9), wt is represented by formula (10), and ht is represented by formula (11).

[0101] yt=(mImageHeight-mImageWidth*mSurfaceHeight / mSurfaceWidth) / 2 Formula (9)

[0102] wt=mImageWidth formula (10)

[0103] ht=mImageHeight-yt; formula (11)

[0104] When selecting texture feature data to be compressed from various texture feature data, the display device can segment each texture feature data according to the data acquisition interval to obtain each segment of data. The interval between each segment of data is SW / DW. For each segment of data, RGB sampling is performed to obtain the texture feature data to be compressed.

[0105] When a display device performs RGB sampling, it can select only the R, G, and B channel data, excluding the A channel data. Taking Xi (corresponding to Ri, Gi, Bi) as an example, Xi is any segment of data. Depending on the data acquisition interval, if the previous segment of data is Xi, then the next segment of data will be Xi + SW / DW.

[0106] For example, for the texture feature data of each pixel, its scaled coordinate resizeLoc can be calculated according to formula (12), and its unscaled coordinate origLoc can be calculated according to formula (13). Among them, origW can be calculated according to formula (14), and origH can be calculated according to formula (15).

[0107] resizeLoc=y*resizeWidth+x formula (12)

[0108] origLoc=origH*origWidth+origW formula (13)

[0109] origW=x*ratioWidth formula (14)

[0110] origH=y*ratioHeight formula (15)

[0111] Wherein, ratioWidth represents the width ratio between the image to be processed and the target image, ratioHeight represents the height ratio between the image to be processed and the target image, resizeWidth represents the width of the target image, resizeHeight represents the height of the target image, origWidth represents the width of the image to be processed, and origHeight represents the height of the image to be processed. In this embodiment, the width of the target image can also be understood as the width of the screen, and the height of the target image can also be understood as the height of the screen.

[0112] The ratioWidth can be determined using the following formula (16), and the ratioHeight can be determined using the following formula (17).

[0113] ratioWidth=origWidth / resizeWidth formula (16)

[0114] ratioHeight=origHeight / resizeHeight formula (17)

[0115] A3. The display device obtains the target image based on the obtained texture feature data to be compressed.

[0116] For example, see Figure 13A and Figure 13B As shown, Figure 13A This is a schematic diagram of the image to be processed. The display device selects the texture feature data to be compressed from various texture feature data according to the data acquisition interval, and then obtains the compressed texture feature data based on the obtained data. Figure 13B The target image shown.

[0117] Operation 3: Since the display device has high requirements for data processing speed, in order to improve the data processing speed of the AI ​​model, in this embodiment of the application, the display device can, after obtaining each texture feature data, input each texture feature data into the joint detection model to obtain each joint of the target object contained in the image to be processed, and mark each joint in the image to be processed to obtain the target image.

[0118] It should be noted that the key point detection model is only an example. In practical applications, it can also be applied to other AI models, such as gesture recognition models and image classification models.

[0119] Since there is a mapping relationship between the storage location of each texture feature data and the pixel, in this embodiment, when annotating each joint point to the image to be processed, the display device can determine the storage location of each joint point in the cache data to be processed based on the compression ratio. Then, based on the determined storage location, the pixel corresponding to each joint point is determined, and thus each joint point is annotated to the image to be processed. In practical applications, annotating each joint point to the image to be processed can be achieved through matrix transformation, which will not be elaborated here.

[0120] For example, taking the target object as a human image, the display device inputs the texture feature data of each object into the joint detection model to obtain 14 joints of the human image contained in the image to be processed, and then marks the 14 joints in the image to be processed to obtain the target image.

[0121] In some embodiments, since the image format corresponding to the cache data to be processed obtained by the display device from the cache is YUV format, while the image format of the input data of the AI ​​model is RGB format, in this embodiment, before inputting the target cache data into the joint detection model, it is necessary to convert the YUV format data into RGB format data.

[0122] Specifically, based on the data conversion relationship between the camera image format and the display device image format, the display device converts each texture feature data into target cache data. The image format corresponding to the target cache data is the display device image format, which is different from the camera image format. Then, the target cache data is input into the joint detection model to obtain each joint of the target object contained in the image to be processed.

[0123] For example, the camera image format is YUV format, and the display device image format is RGB format. The display device converts each texture feature data into target cache data in RGB format according to the data conversion relationship between YUV and RGB formats. Then, the target cache data is input into the joint detection model to obtain each joint of the target object contained in the image to be processed.

[0124] Operation 4: The display device, for the target cropping region contained in the image to be processed, determines the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region from various texture feature data, determines the compression ratio according to the size of the image to be processed, and obtains the target image according to the compression ratio and the texture feature data to be cropped.

[0125] Since the cropping process of the image to be processed by the display device is the same as the cropping process described above, and the compression process of the image to be processed is the same as the compression process described above, it will not be repeated here.

[0126] It should be noted that the above operations 1, 2, and 3 are only examples. Operations 1, 2, and 3 can be combined and used according to the actual application scenario. This application does not impose any restrictions on this. For example, operation 4 is a combination of operation 1 and operation 2.

[0127] The present application will now be described with reference to a specific embodiment. (See attached document.) Figure 14 As shown, the image processing flow is as follows.

[0128] S1401, The display device acquires the collected image A to be processed, and the display format of the image A to be processed is 1080P.

[0129] S1402, The display device obtains the cache data 1 corresponding to the image A to be processed, and caches the cache data 1 corresponding to the image A to be processed according to the mapping relationship between pixels and storage locations.

[0130] The process of obtaining cache data 1 corresponding to the image to be processed A is the same as the process of obtaining cache data to be processed described above, and will not be repeated here.

[0131] S1403. The display device responds to the image processing operation and obtains the texture feature data corresponding to each pixel in the image to be processed, according to the mapping relationship.

[0132] The process of obtaining each texture feature data is the same as that of obtaining texture feature data in the above text, and will not be repeated here.

[0133] S1404. The display device obtains cache data 2 corresponding to the 720P image based on cache data 1.

[0134] Specifically, the display device can determine the compression ratio for converting the 1080P image to a 720P image based on the size of the image A to be processed (1080P). Then, based on the compression ratio and the texture feature data contained in cache data 1, it obtains the various texture feature data to be compressed, which serves as cache data 2 corresponding to the 720P image. Since the process of obtaining the texture feature data to be compressed is the same as the process described above, it will not be repeated here.

[0135] If the image format of cached data 1 is YUV format, then the display device can also convert the image format of each texture feature data to be compressed according to the data conversion relationship between YUV format and RGB format, and use each texture feature data to be compressed in RGB format as cached data 2 corresponding to the 720P image.

[0136] S1405, the display device inputs cached data 2 into the target object detection model to determine the target cropping region contained in the image to be processed. For example, the target object can be a person image, and the target cropping region can be the region containing the person image.

[0137] S1406: The display device retrieves various texture feature data from cached data 2 and determines the texture feature data to be clipped corresponding to the target pixels contained in the target clipping region. Since S1406 is the same as the process for determining the texture feature data to be clipped described above, it will not be repeated here.

[0138] S1407: The display device inputs the texture feature data to be clipped into the joint detection model to obtain the joints of the target object. Since S1407 is the same as the process of determining the joints of the target object mentioned above, it will not be described again here.

[0139] S1408: The display device annotates each joint point onto the image to be processed, obtaining the target image. This target image is a 1080P image with each joint point annotated. Since S1408 is the same process as annotating each joint point onto the image to be processed to obtain the target image as described above, it will not be repeated here.

[0140] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the discussion in some embodiments above is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific application considerations.

[0141] Based on the same inventive concept, this application provides an image processing apparatus, see below. Figure 15 As shown, it includes at least: a cache unit 1501 and a processing unit 1502.

[0142] The caching unit 1501 is used to render the acquired image to be processed to obtain the texture information of the image to be processed, and to obtain the cache data to be processed corresponding to the image to be processed based on the texture information, and to cache the cache data to be processed based on the mapping relationship between pixels and storage locations. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel.

[0143] The processing unit 1502 is configured to, in response to an image processing operation, determine the storage location of each texture feature data in the cache data to be processed according to the mapping relationship, acquire each texture feature data according to the determined storage location, and obtain a target image according to the texture feature data.

[0144] In some embodiments, when obtaining the target image based on the respective texture feature data, the processing unit 1502 is configured to perform at least one of the following operations:

[0145] For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the texture feature data, and the target image is obtained based on the texture feature data to be cropped.

[0146] Based on the size of the image to be processed, a compression ratio is determined, and a target image is obtained based on the compression ratio and the various texture feature data. For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the various texture feature data, and a compression ratio is determined based on the size of the image to be processed. The target image is then obtained based on the compression ratio and the texture feature data to be cropped.

[0147] In some embodiments, when obtaining the target image based on the compression ratio and the respective texture feature data, the processing unit 1502 is used to:

[0148] Based on the compression ratio, a data acquisition interval is determined; based on the data acquisition interval, texture feature data to be compressed is selected from each texture feature data; based on the obtained texture feature data to be compressed, the target image is obtained.

[0149] In some embodiments, when selecting texture feature data to be compressed from the various texture feature data according to the data acquisition interval, the processing unit 1502 is used to:

[0150] Based on the compression ratio, a starting position is determined, whereby the starting position represents the position of the center point of the image to be processed in the buffer data to be processed. Starting from the starting position, based on the data acquisition interval, texture feature data to be compressed is selected from each texture feature data.

[0151] In some embodiments, when obtaining the target image based on the respective texture feature data, the processing unit 1502 is used to:

[0152] The texture feature data are input into the joint detection model to obtain the joints of the target object contained in the image to be processed; the joints are then labeled in the image to be processed to obtain the target image.

[0153] In some embodiments, the image format corresponding to the cached data to be processed is a camera image format;

[0154] When the texture feature data is input into the joint detection model to obtain the joints of the target object contained in the image to be processed, the processing unit 1501 is used to:

[0155] Based on the data conversion relationship between the camera image format and the display device image format, the texture feature data is converted into target cache data, and the image format corresponding to the target cache data is the display device image format, which is different from the camera image format. The target cache data is then input into the joint detection model to obtain the joints of the target object contained in the image to be processed.

[0156] Based on the same inventive concept, embodiments of this application provide a storage medium that, when the instructions in the storage medium are executed by a processor, enables the processor to execute any of the methods implemented by the image processing device in the above process.

[0157] In summary, in this embodiment of the application, by caching the data to be processed based on the mapping relationship between pixels and storage locations during the rendering process of the image to be processed, and then, in response to the image processing operation, the texture feature data corresponding to each pixel in the image to be processed can be quickly obtained based on the above mapping relationship. In this way, the texture feature data is processed directly instead of the pixels, which reduces the amount of computation, thereby reducing the CPU usage and improving the image processing efficiency.

[0158] For system / device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0159] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0160] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0162] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0164] Although preferred embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this disclosure.

[0165] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A display device, characterized in that, include: A monitor is used to display the user interface. The controller is configured as follows: The acquired image to be processed is rendered to obtain the texture information of the image to be processed. Based on the texture information, the cache data to be processed corresponding to the image to be processed is obtained. Based on the mapping relationship between pixels and storage locations, the cache data to be processed is cached. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel. The cached data to be processed is converted into an array and stored; wherein the data in the cached data to be processed stored in the array is continuous, and the correspondence between rows and columns in the array reflects the mapping relationship between pixels and storage locations; In response to the image processing operation, the storage location of each texture feature data in the cache data to be processed is determined according to the mapping relationship, and the texture feature data is obtained according to the determined storage location, and the target image is obtained according to the texture feature data.

2. The device as described in claim 1, characterized in that, When obtaining the target image based on the various texture feature data, the controller is further configured to perform at least one of the following operations: For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the texture feature data, and the target image is obtained based on the texture feature data to be cropped. Based on the size of the image to be processed, a compression ratio is determined, and based on the compression ratio and the various texture feature data, a target image is obtained; For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the various texture feature data, and the compression ratio is determined according to the size of the image to be processed, and the target image is obtained according to the compression ratio and the texture feature data to be cropped.

3. The device as described in claim 2, characterized in that, When obtaining the target image based on the compression ratio and the various texture feature data, the controller is further configured to: The data acquisition interval is determined based on the compression ratio. Based on the data acquisition interval, select the texture feature data to be compressed from each texture feature data; The target image is obtained based on the obtained texture feature data to be compressed.

4. The device as described in claim 3, characterized in that, When selecting texture feature data to be compressed from the various texture feature data according to the data acquisition interval, the controller is further configured to: The starting position is determined according to the compression ratio, and the starting position represents the position of the center point of the image to be processed in the buffer data to be processed; Starting from the initial position, according to the data acquisition interval, texture feature data to be compressed is selected from each texture feature data.

5. The device as described in any one of claims 1-4, characterized in that, When obtaining the target image based on the various texture feature data, the controller is further configured to: The texture feature data are input into the joint detection model to obtain the joints of the target object contained in the image to be processed. The various joints are labeled onto the image to be processed to obtain the target image.

6. The device as described in claim 5, characterized in that, The image format corresponding to the cached data to be processed is the camera image format; When the various texture feature data are input into the joint detection model to obtain the joints of the target object contained in the image to be processed, the controller is further configured to: Based on the data conversion relationship between the camera image format and the display device image format, each texture feature data is converted into target cache data. The image format corresponding to the target cache data is the display device image format, and the camera image format is different from the display device image format. The target cache data is input into the joint detection model to obtain each joint of the target object contained in the image to be processed.

7. An image processing method, characterized in that, include: The acquired image to be processed is rendered to obtain the texture information of the image to be processed. Based on the texture information, the cache data to be processed corresponding to the image to be processed is obtained. Based on the mapping relationship between pixels and storage locations, the cache data to be processed is cached. The image to be processed contains each pixel, and the cache data to be processed stores the texture feature data corresponding to each pixel. The cached data to be processed is converted into an array and stored; wherein the data in the cached data to be processed stored in the array is continuous, and the correspondence between rows and columns in the array reflects the mapping relationship between pixels and storage locations; In response to the image processing operation, the storage location of each texture feature data in the cache data to be processed is determined according to the mapping relationship, and the texture feature data is obtained according to the determined storage location, and the target image is obtained according to the texture feature data.

8. The method as described in claim 7, characterized in that, Obtaining the target image based on the various texture feature data includes at least one of the following operations: For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the texture feature data, and the target image is obtained based on the texture feature data to be cropped. Based on the size of the image to be processed, a compression ratio is determined, and based on the compression ratio and the various texture feature data, a target image is obtained; For the target cropping region contained in the image to be processed, the texture feature data to be cropped corresponding to the target pixels contained in the target cropping region is determined from the various texture feature data, and the compression ratio is determined according to the size of the image to be processed, and the target image is obtained according to the compression ratio and the texture feature data to be cropped.

9. The method as described in claim 8, characterized in that, The step of obtaining the target image based on the compression ratio and the various texture feature data includes: The data acquisition interval is determined based on the compression ratio. Based on the data acquisition interval, select the texture feature data to be compressed from each texture feature data; The target image is obtained based on the obtained texture feature data to be compressed.

10. The method as described in claim 9, characterized in that, The step of selecting texture feature data to be compressed from the various texture feature data according to the data acquisition interval includes: The starting position is determined according to the compression ratio, and the starting position represents the position of the center point of the image to be processed in the buffer data to be processed; Starting from the initial position, according to the data acquisition interval, and following the acquisition order from the center outwards, the texture feature data to be compressed is selected from each texture feature data.

Images