Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an image processing method and device, electronic equipment and a storage medium, the method is applied to the electronic equipment arranged in a smart classroom scene, and the electronic equipment supports heterogeneous processing of a central processing unit (CPU) and a graphics processing unit (GPU). The method comprises the steps of determining image data in a shared memory supporting sharing of a CPU and a GPU and a to-be-converted format of the image data; determining whether the format to be converted belongs to a preset format set; when it is determined that the format to be converted belongs to the preset format set, the CPU performs conversion processing on a Y component of the image data to obtain a target Y component signal; the GPU performs conversion processing on the U component and the V component of the image data to obtain a target U component signal and a target V component signal; and obtaining image data in a target format based on the obtained target Y component signal, the target U component signal and the V component signal.

Description

Background Art

[0002] At present, with the continuous development of technology, a new type of classroom has gradually emerged, which is constructed by means of Internet of Things technology, cloud computing technology, intelligent technology, etc. This new type of classroom is, for example, called a smart classroom. In the smart classroom scenario, there are tangible physical spaces and intangible digital spaces, and various intelligent devices are used to assist in presenting teaching content. Specifically, the teaching content can be presented in the form of videos.

[0003] In the related art, in the smart classroom scenario, video images are compressed using H.264 / H.265 and then transmitted over the network. To save transmission bandwidth overhead, the encoder hardware of electronic devices in the smart classroom scenario only supports the YUV420 format. Also, the original image format is generally RGB888 or YUV444. Therefore, it is necessary to convert the image format from RGB888 or YUV444 to YUV420 before sending it to the encoder hardware of the electronic device for encoding.

[0004] However, currently, the color format conversion hardware devices provided by different chip manufacturers do not comprehensively support color formats, color spaces, etc., which may result in the inability to convert the original image format from RGB888 or YUV444 to YUV420 or a very poor conversion effect. Summary of the Invention

[0005] An embodiment of the present application provides an image processing method, apparatus, electronic device, and storage medium for solving the technical problem that the electronic device in the smart classroom scenario cannot convert the original image format from RGB888 or YUV444 to YUV420 or has a very poor conversion effect.

[0006] On the one hand, an embodiment of the present application provides an image processing method, which is applied to an electronic device set in a smart classroom scenario. The electronic device supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU). The method includes:

[0007] Determine the image data in the shared memory supported by the CPU and GPU and the format to be converted of the image data;

[0008] Determine whether the format to be converted belongs to a preset format set, where the preset format set includes the formats of image data that support heterogeneous processing by the central processing unit (CPU) and the graphics processing unit (GPU);

[0009] When it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and V component of the image data to obtain a target U component signal and a target V component signal; and based on the obtained target Y component signal, target U component signal, and target V component signal, the image data in the target format is obtained.

[0010] In a possible implementation manner, the method further includes:

[0011] When it is determined that the format to be converted does not belong to the preset format set, the graphics processing unit (GPU) of the image processor performs the following operations:

[0012] Create an Open Computing Language (OPENCL) GPU kernel;

[0013] Configure a color conversion matrix according to the color space and color range of the image data, select a working function corresponding to the target format, and load the working function into the Open Computing Language (OPENCL) GPU kernel; the working function is used to perform a color conversion matrix multiplication operation on the pixels in the image data;

[0014] Configure an OPENCL work item range according to the pixel rate of the pixels in the image data, and start an OPENCL work queue and the working function configured with the OPENCL work item range to process the image data, so as to obtain a target Y component signal, a target U component signal, and a target V component signal corresponding to each pixel in the image data;

[0015] Obtain the image data in the target format according to the target Y component signal, target U component signal, and target V component signal corresponding to all the images in the image data.

[0016] In a possible implementation manner, after creating the Open Computing Language (OPENCL) GPU kernel, the method further includes:

[0017] Load an OPENCL Kernel computing model into the Open Computing Language (OPENCL) GPU kernel; the OPENCL Kernel computing model is used to perform encryption processing when converting the image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

[0018] In a possible implementation manner, the preset format set includes YUV444 format and YUV422 format.

[0019] On the one hand, an embodiment of the present application provides an image processing device, which is applied to an electronic device set in a smart classroom scenario. The electronic device supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU). The device includes:

[0020] A first determination unit, configured to determine image data in a shared memory supported by the CPU and the GPU and a format to be converted of the image data;

[0021] A second determination unit, configured to determine whether the format to be converted belongs to a preset format set, where the preset format set includes formats of image data that support heterogeneous processing by the central processing unit (CPU) and the graphics processing unit (GPU);

[0022] An obtaining unit, configured to when it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and the V component of the image data to obtain a target U component signal and a V component signal; and based on the obtained target Y component signal, target U component signal, and V component signal, obtain the image data in the target format.

[0023] In a possible implementation manner, the obtaining unit is further configured to:

[0024] When it is determined that the format to be converted does not belong to the preset format set, the image processor GPU performs the following operations:

[0025] Create an Open Computing Language (OPENCL) GPU kernel;

[0026] Configure a color conversion matrix according to the color space and color range of the image data, select a working function corresponding to the target format, and load the working function into the Open Computing Language (OPENCL) GPU kernel; the working function is used to perform color conversion matrix multiplication operations on pixels in the image data;

[0027] Configure an OPENCL work item range according to the pixel rate of pixels in the image data, and start an OPENCL work queue and the working function configured with the OPENCL work item range to process the image data, so as to obtain a target Y component signal, a target U component signal, and a target V component signal corresponding to each pixel in the image data;

[0028] Obtain the image data in the target format according to the target Y component signal, target U component signal, and target V component signal corresponding to all images in the image data.

[0029] In a possible implementation, after creating an Open Computing Language (OpenCL) GPU kernel, the obtaining unit is further configured to:

[0030] Load an OpenCL Kernel computing model into the OpenCL GPU kernel; the OpenCL Kernel computing model is used to perform encryption processing when converting the image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

[0031] In a possible implementation, the preset format set includes YUV444 format and YUV422 format.

[0032] On the one hand, an electronic device provided in an embodiment of the present application includes a processor and a memory. The memory stores program code. When the program code is executed by the processor, the processor is caused to execute the above image processing method.

[0033] On the one hand, an embodiment of the present application further provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the above image processing method.

[0034] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which includes program code. When the program product runs on an electronic device, the program code is used to cause the electronic device to execute the steps of any one of the above image processing methods.

[0035] The beneficial effects of the present application are as follows:

[0036] In an embodiment of the present application, an electronic device is set in a smart classroom scenario, and the electronic device supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU). In this way, the electronic device can determine the image data in the shared memory supported by the CPU and the GPU and the format to be converted of the image data. Further, when the electronic device can determine whether the format to be converted belongs to a preset format set, the preset format set includes the formats of image data that support heterogeneous processing by the central processing unit (CPU) and the graphics processing unit (GPU); thus, when it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and the V component of the image data to obtain a target U component signal and a target V component signal; and based on the obtained target Y component signal, target U component signal, and target V component signal, image data in the target format is obtained. That is to say, in the embodiment of the present application, the electronic device coordinates CPU / GPU resources for heterogeneous computing, satisfying image processing in various formats while ensuring the lowest latency and improving the user experience.

[0037] Other features and advantages of the present application will be described in the following description, and some of them will be obvious from the description, or can be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written description, claims, and drawings. Brief Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0039] Figure 1 It is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0040] Figure 2 It is a schematic flowchart of an image processing method in an embodiment of the present application;

[0041] Figure 3 It is another flowchart of an image processing method in an embodiment of the present application;

[0042] Figure 4 It is a schematic diagram of the composition structure of an image processing device in an embodiment of the present application;

[0043] Figure 5 It is a schematic diagram of a structure of an electronic device in an embodiment of the present application. Detailed implementation manners

[0044] To make the objectives, technical solutions and advantages of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part rather than all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application. Without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other arbitrarily. And although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a sequence different from that here.

[0045] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here.

[0046] The term "exemplary" used in the present application means "serving as an example, an embodiment or an illustration". Any embodiment described as "exemplary" does not necessarily need to be construed as superior to or better than other embodiments.

[0047] To facilitate the understanding of the technical solutions provided in the embodiments of the present application, some key terms related to the embodiments of the present application are explained here first:

[0048] 1. H.264, also known as advanced video coding (AVC) or MPEG-4 Part 10, is the most widely used video compression standard in the industry currently. H.264 uses block-oriented coding.

[0049] 2. H.265, also known as High Efficiency Video Coding (HEVC) or MPEG-H part 2, is a video compression standard. It uses the coding of coding tree units (CTUs), that is, the sizes of the video frames decomposed by H.264 and H.265 are different.

[0050] 3. YUV (Y'CbCr) is a color space model, commonly seen in video coding and still images. Contrary to the RGB format (Red - Green - Blue), YUV is respectively represented by a "luminance" component (Luminance or Luma) called Y (equivalent to grayscale) and two "chrominance" components (Chrominance or Chroma) called U (blue projection Cb) and V (red projection Cr).

[0051] The sampling rate of the chrominance channels (UV) can be lower than that of the luminance channel (Y) without significantly degrading the perceived quality. A notation called "A:B:C" is used to describe the frequencies of U and V relative to the Y sampling.

[0052] The YUV444 format, where the frequencies of U and V relative to the Y sampling are 4:4:4, means that the sampling rate of the chrominance (UV) channels is not reduced, and each Y component corresponds to a set of UV components.

[0053] The YUV422 format, where the frequencies of U and V relative to the Y sampling are 4:2:2, means 2:1 horizontal downsampling without vertical downsampling. Every two Y components share a set of UV components.

[0054] The YUV420 format, where the frequencies of U and V relative to the Y sampling are 4:2:0, means 2:1 horizontal downsampling and 2:1 vertical downsampling. Every four Y components share a set of UV components.

[0055] 4. RGB is a color space model, where R, G, and B respectively refer to Red, Green, and Blue. The RGB888 format means that each of the three color channels of red, green, and blue occupies 8 bits, for a total of 24 bits, and 2 to the 24th power is 16,777,216 colors.

[0056] 5. The conversion between the RGB color space and the YUV color space is performed based on existing standards, such as the three standards of ITU-R BT.601 (standard definition), ITU-R BT.709 (high definition), and ITU-R BT.2020 (ultra high definition). Different standards have different conversion formulas, and different color ranges also need to be distinguished. Color Range is used to specify the value range of RGB components, which can be divided into the full range (Full Range) with a value range of 0 to 255 and the limited range (Limited Range) with a value range of 16 to 235.

[0057] Currently, the color format conversion hardware devices provided by different chip manufacturers (such as dedicated 2D image processing modules or image signal processing modules (ISP) built into the chip) do not support color formats and color spaces comprehensively, which may result in the inability to convert the format of the original image from RGB888 or YUV444 to YUV420 or a very poor conversion effect.

[0058] In view of this, the present application provides an image processing method, apparatus, electronic device, and storage medium. Since the electronic device is arranged in a smart classroom scenario and supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU), the electronic device can determine the image data in the shared memory supported by the CPU and GPU and the format to be converted of the image data. Further, when the electronic device can determine whether the format to be converted belongs to a preset format set, the preset format set includes the formats of image data that support heterogeneous processing by the central processing unit (CPU) and the graphics processing unit (GPU); thus, when it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and V component of the image data to obtain a target U component signal and a target V component signal; and based on the obtained target Y component signal, target U component signal, and target V component signal, image data in the target format is obtained. That is to say, in the embodiments of the present application, the electronic device coordinates CPU / GPU resources for heterogeneous computing, satisfying image processing in various formats while ensuring the lowest latency and improving the user experience.

[0059] After introducing the design concept of the embodiments of the present application, the following briefly introduces the application scenarios applicable to the technical solutions of the embodiments of the present application. It should be noted that the following introduced application scenarios are only used to illustrate the embodiments of the present application rather than to limit them. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0060] The solution provided by the embodiments of the present application can be applied to an image processing scenario that requires format conversion of image data, such as the aforementioned smart classroom scenario.

[0061] Refer to Figure 1 As shown, it is a schematic diagram of an application scenario in the embodiments of the present application. This application scenario includes at least a terminal device 110 and a server 130, and the operation interface 120 can be logged in through the terminal device 110. The number of terminal devices 110 can be one or more, and the number of servers 130 can also be one or more. The present application does not make a specific limitation on the number of terminal devices 110 and servers 130. The terminal device 110 and the server 130 can communicate through a communication network.

[0062] In the embodiments of the present application, a video playback platform is pre-installed in the terminal device 110, and users can complete operations such as viewing, collecting, and marking videos through this platform. The terminal device 110 can be, but is not limited to, a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a smart home device, a smart voice interaction device, a vehicle-mounted terminal, etc. Among them, each terminal device 101 can include one or more processors, a memory, and an I / O interface for interacting with the server, etc., Figure 1 which is not shown in the figure.

[0063] The server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0064] Among them, a direct or indirect communication connection can be established between the terminal device 110 and the server 130, and between each terminal device 110 through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network. Of course, it can also be other possible networks, and the embodiments of the present application do not limit this.

[0065] The image processing method in the embodiments of the present application can be executed by the terminal device 110, or by the server 130, or by the interaction between the terminal device 110 and the server 130.

[0066] Taking the server 130 executing the image processing method in the embodiments of the present application as an example, the method includes the following steps:

[0067] When the server 130 determines the image data and the to-be-converted format of the image data in the shared memory that supports CPU and GPU sharing; determines whether the to-be-converted format belongs to a preset format set, and the preset format set includes the formats of image data that support heterogeneous processing of the central processing unit (CPU) and the graphics processing unit (GPU); when it is determined that the to-be-converted format does not belong to the preset format set, perform a matrix multiplication operation for image color conversion on the image data based on the multiplier in the graphics processing unit (GPU) to convert the image data from the to-be-converted format to the target format, and obtain the image data in the target format, so that the image data in the target format can be encoded and decoded, etc., to present high-quality image data to the user.

[0068] Of course, the method provided by the embodiments of the present application is not limited to being used forFigure 1 In the application scenario shown, it can also be used in other possible application scenarios, and the embodiments of the present application do not limit this. For Figure 1 the functions that can be achieved by each device in the application scenario shown will be described together in the subsequent method embodiments, and will not be elaborated here too much.

[0069] To further illustrate the technical solutions provided by the embodiments of the present application, the following will be described in detail in combination with the accompanying drawings and specific implementation manners. Although the embodiments of the present application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or non-creative labor. In steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiments of the present application. When the method is actually processed or executed by the device, it can be executed in the order shown in the embodiments or drawings or executed in parallel.

[0070] Please refer to Figure 2 , Figure 2 which shows a flowchart of an image processing method provided by an embodiment of the present application. The process of this method can be executed by an electronic device, and this electronic device can be Figure 1 the terminal device 110 and / or the server 130 in Figure 2 . As shown in

[0071] Step 201: Determine the image data in the shared memory that supports CPU and GPU sharing and the format to be converted for the image data.

[0072] In the embodiments of the present application, the CPU can convert the memory buffer that originally stored RGB image data (abbreviated as rgb_buff) and received and output YUV image data (abbreviated as yuv_buff) into a buffer that can be accessed by both the CPU and the OPENCL GPU, that is, a shared buffer area. For example, Dma_buffer, and Dma_buffer is a buffer area that allows sharing between the CPU and other subsystems or various input / output peripherals. In this way, the electronic device can determine the shared memory Dma_buffer that supports CPU and GPU sharing, and then obtain the image data from this shared memory. The image data in this shared memory is, for example, the image output by the kernel driver V4L2 (Video for linux2) of the video device in linux for collecting High-Definition Multimedia Interface (HDMI-IN).

[0073] In the embodiment of the present application, the electronic device can also determine the format to be converted of the image data. It should be noted that the format to be converted in the embodiment of the present application can be understood as the format in the corresponding color space.

[0074] Step 202: Determine whether the format to be converted belongs to a preset format set, where the preset format set includes formats of image data that support heterogeneous processing of the central processing unit (CPU) and the graphics processing unit (GPU).

[0075] In the embodiment of the present application, after the electronic device determines the format to be converted of the image data, it can further determine whether the format to be converted belongs to a preset format set. For example, the preset format set includes the YUV444 format and the YUV422 format. That is, it is determined whether the CPU and the GPU need to jointly implement the conversion of the format to be converted.

[0076] Step 203: When it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and the V component of the image data to obtain a target U component signal and a target V component signal; and based on the obtained target Y component signal, target U component signal, and target V component signal, image data in the target format is obtained.

[0077] In the embodiment of the present application, considering that the GPU performance on some system on chip (SoC) is not strong, and the Y components of YUV444 / YUV422 and YUV420 are otherwise unchanged, according to the characteristics of the image Y component, the CPU can transfer the Y component of YUV444 / YUV422 to the Y of YUV420 at a lower cost. Therefore, the CPU can directly perform a direct memory access (DMA) operation on the target buffer for the Y component in the image data.

[0078] In the embodiment of the present application, when the electronic device determines that the format to be converted does not belong to the preset format set, that is, when it is determined that the conversion of the format to be converted can be performed only by the graphics processing unit (GPU) alone, the graphics processing unit (GPU) can obtain the image data in the target format by, but not limited to, the following steps:

[0079] Step A: Create an Open Computing Language (OpenCL) GPU kernel.

[0080] In an embodiment of the present application, after creating an Open Computing Language (OPENCL) GPU kernel, an electronic device can also load an OPENCL Kernel computing model into the OPENCL GPU kernel; the OPENCL Kernel computing model is used to perform encryption processing when converting image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

[0081] Step B: Configure a color conversion matrix according to the color space and color range of the image data, select a working function corresponding to the target format, and load the working function into the Open Computing Language (OPENCL) GPU kernel; the working function is used to perform a color conversion matrix multiplication operation on the pixels in the image data.

[0082] In an embodiment of the present application, the GPU can load and submit a corresponding working function (such as the language of kernels on an OPENCL device) into the OPENCL GPU kernel. The working function can be understood as an execution conversion model, and the execution conversion model can be understood as a small work item, and each work item is used to perform the work of converting an RGB pixel to YUV.

[0083] Step C: Configure the OPENCL work item range according to the pixel rate of the pixels in the image data, and start the OPENCL work queue and working function with the configured OPENCL work item range to process the image data, so as to obtain the target Y component signal, target U component signal, and V component signal corresponding to each pixel in the image data;

[0084] Step D: Obtain the image data in the target format according to the target Y component signal, target U component signal, and V component signal corresponding to all images in the image data;

[0085] In an embodiment of the present application, the GPU creates a two-dimensional work item set. Each work item has a coordinate (gx, gy) in the global two-dimensional index space, and the size of the global index space is (Gx, Gy). The length and width information of the RGB image of the image data, the information of rgb_buff storing the input RGB image data, the information of yuv_buff for storing the output YUV image data, and the corresponding color conversion matrix (matrix) are used as parameters to start the OPENCL device to execute the foregoing work items. The work items will be executed concurrently on the processing units of a computing unit, extract the R / G / B color data of the RGB image pixel lattice from rgb_buff, then perform a matrix multiplication operation with matrix to obtain the Y / U / V component signals and save them to yuv_buff, and store the Y / U / V component signals in the NV12 format as complete YUV420 image data. Thus, the image data in the target format is obtained.

[0086] In the embodiments of the present application, compared with the traditional fixed processing method of hardware converters, in the embodiments of the present application, different custom algorithms can be dynamically loaded through OPENCL. While maintaining the performance level, image watermarks can be added. Specifically, the OPENCL GPU computing model can process the U and V components during conversion and perform reverse restoration of the data after decoding at the target end. In this way, only the proprietary device can display the image normally, and the image displayed after other devices steal the stream becomes abnormal.

[0087] Based on the same inventive concept, the embodiments of the present application also provide an image processing device. As Figure 3 shown, it is a schematic structural diagram of the image processing device 300, which may include:

[0088] A first determination unit 301, configured to determine the image data in the shared memory that supports CPU and GPU sharing and the format to be converted of the image data;

[0089] A second determination unit 302, configured to determine whether the format to be converted belongs to a preset format set, where the preset format set includes the formats of image data that support heterogeneous processing of the central processing unit (CPU) and the graphics processing unit (GPU);

[0090] An obtaining unit 303, configured to, when it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and V component of the image data to obtain a target U component signal and a V component signal; and based on the obtained target Y component signal, target U component signal, and V component signal, obtain the image data in the target format.

[0091] In a possible implementation manner, the obtaining unit 303 is further configured to:

[0092] When it is determined that the format to be converted does not belong to the preset format set, the image processor GPU performs the following operations:

[0093] Create an Open Computing Language (OPENCL) GPU kernel;

[0094] Configure a color conversion matrix according to the color space and color range of the image data, select a working function corresponding to the target format, and load the working function into the Open Computing Language (OPENCL) GPU kernel; the working function is used to perform color conversion matrix multiplication operations on the pixels in the image data;

[0095] Configure the OPENCL work item range according to the pixel rate of the pixels in the image data, and start the OPENCL work queue and the work function configured with the OPENCL work item range to process the image data, so as to obtain the target Y component signal, target U component signal, and V component signal corresponding to each pixel in the image data;

[0096] Obtain the image data in the target format according to the target Y component signal, target U component signal, and V component signal corresponding to all the images in the image data.

[0097] In a possible implementation manner, after creating the OpenCL GPU kernel, the obtaining unit 303 is further configured to:

[0098] Load the OpenCL Kernel computing model into the OpenCL GPU kernel; the OpenCL Kernel computing model is used to perform encryption processing when converting the image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

[0099] In a possible implementation manner, the preset format set includes the YUV444 format and the YUV422 format.

[0100] For the convenience of description, the above parts are divided into respective modules (or units) according to functions and described separately. Of course, when implementing the present application, the functions of the respective modules (or units) can be implemented in the same or multiple software or hardware.

[0101] After introducing the image processing method and device of the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0102] Those skilled in the art of the present technology can understand that various aspects of the present application can be implemented as a system, method, or program product. Therefore, various aspects of the present application can be specifically implemented in the following forms, namely: a complete hardware implementation manner, a complete software implementation manner (including firmware, microcode, etc.), or an implementation manner combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0103] Based on the same inventive concept as the above method embodiment, an electronic device is further provided in the embodiment of the present application. In one embodiment, the electronic device can be a server, such as Figure 1 the server 130 shown. In this embodiment, the structure of the electronic device can be as Figure 4 shown, including a memory 401, a communication module 403, and one or more processors 402.

[0104] A memory 401 for storing a computer program executed by a processor 402. The memory 401 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and programs required to run a video playback function, etc.; the data storage area may store various image data and operation instruction sets, etc.

[0105] The memory 401 may be a volatile memory, such as a random-access memory (RAM); the memory 401 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), or the memory 401 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 401 may be a combination of the above memories.

[0106] The processor 402 may include one or more central processing units (CPUs) or be a digital processing unit, etc. The processor 402 is used to implement the above image processing method when calling the computer program stored in the memory 401.

[0107] The communication module 403 is used to communicate with other electronic devices. If the electronic device is a server, the server can receive image data sent by other electronic devices through the communication module 403.

[0108] In the embodiments of the present application, the specific connection medium between the above-mentioned memory 401, communication module 403, and processor 402 is not limited. In the embodiments of the present disclosure Figure 4 it is shown that the memory 401 and the processor 402 are connected through a bus 404. The bus 404 is represented by a thick line in Figure 4 The connection manner between other components is only for illustrative purposes and is not to be taken as a limitation. The bus 404 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 4 only a thick line is used to represent it in

[0109] In another embodiment, the electronic device may be any electronic device such as a mobile phone, a tablet computer, a POS (Point of Sales), a vehicle-mounted computer, a smart wearable device, a PC, etc. The electronic device may also be Figure 1 the terminal device 110 shown in

[0110] Figure 5 The structural block diagram of an electronic device provided by an embodiment of the present application is shown. As Figure 5 shown, the electronic device includes components such as a Radio Frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (WiFi) module 570, a processor 580, etc. Those skilled in the art can understand that Figure 5 the structure of the electronic device shown in

[0111] does not limit the electronic device, and may include more or fewer components than shown, or combine some components, or have different component arrangements. Figure 5 The following specifically introduces each component of the electronic device:

[0112] The RF circuit 510 can be used for receiving and sending signals during information reception or call processes. In particular, the obtained signals or information can be sent to the processor 580 for processing.

[0113] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and device in the embodiment of the present application. The processor 580 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 520, such as the image processing method provided in the embodiment of the present application. The memory 520 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs of at least one application, etc.; the data storage area can store data created according to the use of the electronic device. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0114] The input unit 530 can be used to receive image data transmitted by other electronic devices, and generate key signal inputs related to user settings and function controls corresponding to the terminal.

[0115] Optionally, the input unit 530 may include a touch panel 531 and other input devices 532.

[0116] Among them, the touch panel 531, also known as a touch screen, can collect touch operations of a user on or near it (such as operations of a test object using any suitable object or accessory such as a finger or a stylus on or near the touch panel 531), and implement corresponding operations according to a preset program, such as an operation of a test object clicking on a shortcut identifier of a function module. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the test object, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented in multiple types such as resistive, capacitive, infrared, and surface acoustic wave.

[0117] Optionally, the other input device 532 may include, but is not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0118] The display unit 540 can be used to display video data or interface information presented to the user and various menus of the electronic device. The display unit 540 is the display system of the electronic device, used to present an interface, such as displaying a desktop, an operation interface of an application, or an operation interface of a live broadcast application.

[0119] The display unit 540 may include a display panel 541. Optionally, the display panel 541 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0120] Furthermore, the touch panel 531 may cover the display panel 541. After the touch panel 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of touch event. Subsequently, the processor 580 provides a corresponding interface output on the display panel 541 according to the type of touch event.

[0121] Although in Figure 5 the touch panel 531 and the display panel 541 are implemented as two independent components to realize the input and input functions of the electronic device, in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the terminal.

[0122] The electronic device may further include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light. The proximity sensor may turn off the backlight of the display panel 541 when the electronic device is moved close to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used in applications for identifying the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. As for other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors that the electronic device may also be configured with, they will not be elaborated here.

[0123] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the object and the electronic device. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, and the speaker 561 converts it into a sound signal for output. On the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and then converted into audio data. After the audio data is output to the processor 580 for processing, it is sent through the RF circuit 510 to, for example, another electronic device, or the audio data is output to the memory 520 for further processing.

[0124] WiFi belongs to short - range wireless transmission technology. The electronic device can help users send and receive emails, browse the web, and access streaming media through the WiFi module 570. It provides users with wireless broadband Internet access. Although Figure 5 the WiFi module 570 is shown, it can be understood that it does not belong to an essential component of the electronic device and can be omitted completely within the scope of not changing the essence of the invention according to needs.

[0125] The processor 580 is the control center of the electronic device. It connects various parts of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 520, and by calling the data stored in the memory 520, it executes various functions of the electronic device and processes data. Optionally, the processor 580 may include one or more processing units. Optionally, the processor 580 may integrate an application processor and a modem processor. Among them, the application processor mainly processes software programs such as the operating system, applications, and functional modules inside the applications, such as the image processing method provided in the embodiments of the present application. The modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 580 either.

[0126] It can be understood,Figure 5 The structure shown is only illustrative, and the electronic device may further include more or fewer components than those shown in Figure 5 , or have a configuration different from that shown in Figure 5 . Figure 5 Each component shown in can be implemented by hardware, software, or a combination thereof.

[0127] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method in the above embodiments.

[0128] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application.

Claims

1. An image processing method, characterized in that, applied to an electronic device set in a smart classroom scenario, the electronic device supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU), and the method includes: Determining image data in a shared memory supported by the CPU and the GPU and a format to be converted of the image data; Determining whether the format to be converted belongs to a preset format set, the preset format set including formats of image data that support heterogeneous processing by the central processing unit (CPU) and the graphics processing unit (GPU); When it is determined that the format to be converted belongs to the preset format set, the central processing unit (CPU) performs conversion processing on the Y component of the image data to obtain a target Y component signal; and, the graphics processing unit (GPU) performs conversion processing on the U component and the V component of the image data to obtain a target U component signal and a V component signal; and based on the obtained target Y component signal, target U component signal, and V component signal, the image data in the target format is obtained.

2. The method according to claim 1, characterized in that, the method further includes: When it is determined that the format to be converted does not belong to the preset format set, the image processing unit (GPU) performs the following operations: Creating an Open Computing Language (OpenCL) GPU kernel; Configuring a color conversion matrix according to the color space and color range of the image data, selecting a working function corresponding to the target format, and loading the working function into the Open Computing Language (OpenCL) GPU kernel; the working function is used to perform color conversion matrix multiplication operations on the pixels in the image data; Configuring an OpenCL work item range according to the pixel rate of the pixels in the image data, and starting an OpenCL work queue and the working function configured with the OpenCL work item range to process the image data, so as to obtain a target Y component signal, a target U component signal, and a V component signal corresponding to each pixel in the image data; Obtaining the image data in the target format according to the target Y component signal, target U component signal, and V component signal corresponding to all the images in the image data.

3. The method according to claim 2, characterized in that, After creating the Open Computing Language (OpenCL) GPU kernel, the method further includes: Loading an OpenCL Kernel computing model into the Open Computing Language (OpenCL) GPU kernel; the OpenCL Kernel computing model is used to perform encryption processing when converting the image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

4. The method according to any one of claims 1-3, characterized in that, the preset format set includes YUV444 format and YUV422 format.

5. An image processing device, characterized in that, applied to an electronic device set in a smart classroom scenario, the electronic device supports heterogeneous processing by a central processing unit (CPU) and a graphics processing unit (GPU), and the device includes: A first determination unit, configured to determine image data in a shared memory that supports sharing between a CPU and a GPU, and a format to be converted of the image data; A second determination unit, configured to determine whether the format to be converted belongs to a preset format set, where the preset format set includes formats of image data that support heterogeneous processing of a central processing unit (CPU) and a graphics processing unit (GPU); An obtaining unit, configured to, when it is determined that the format to be converted belongs to the preset format set, the CPU performs a conversion process on a Y component of the image data to obtain a target Y component signal; and the GPU performs a conversion process on U and V components of the image data to obtain a target U component signal and a target V component signal; and based on the obtained target Y component signal, target U component signal, and target V component signal, obtain the image data in a target format.

6. The apparatus according to claim 5, wherein the obtaining unit is further configured to: when it is determined that the format to be converted does not belong to the preset format set, the GPU performs the following operations: create an Open Computing Language (OPENCL) GPU kernel; configure a color conversion matrix according to a color space and a color range of the image data, select a working function corresponding to a target format, and load the working function into the OPENCL GPU kernel; the working function is used to perform a color conversion matrix multiplication operation on pixels in the image data; configure an OPENCL work item range according to a pixel rate of pixels in the image data, and start an OPENCL work queue and the working function configured with the OPENCL work item range to process the image data, so as to obtain a target Y component signal, a target U component signal, and a target V component signal corresponding to each pixel in the image data; obtain the image data in the target format according to the target Y component signal, the target U component signal, and the target V component signal corresponding to all images in the image data.

7. The apparatus according to claim 6, wherein after creating the OPENCL GPU kernel, the obtaining unit is further configured to: load an OPENCL Kernel computing model into the OPENCL GPU kernel; the OPENCL Kernel computing model is used to perform an encryption process when converting the image data, so that the obtained image data in the target format can only be decoded and played by the electronic device.

8. The apparatus according to any one of claims 5-7, wherein the preset format set includes YUV444 format and YUV422 format.

9. An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1-4 are implemented.

10. A computer-readable storage medium, wherein It includes program code which, when the program product runs on an electronic device, is used to cause the electronic device to execute the steps of any one of the methods recited in claims 1 to 4.

Images