Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of image processing, and the disclosed image processing method comprises the steps: obtaining a current to-be-processed image frame; determining a target type of the to-be-processed image frame, the target type of the image frame being a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame; in response to the target type of the to-be-processed image frame being the second type, determining difference data between the to-be-processed image frame and a corresponding target reference frame and first position information of the position of the difference data in the to-be-processed image frame; target information containing the first position information is transmitted to an image coding device, so that the image coding device obtains the difference data based on the target information, and a coding result corresponding to the to-be-processed image frame is generated based on the difference data.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, device and electronic device. Background Art

[0002] Currently, when image data needs to be encoded in an electronic device, the image data to be encoded is usually obtained first, and then the image data is hard-coded through a graphics card. For example, when recording or sharing the screen, the display data of the display screen is first obtained and then the display data is hard-coded through the graphics card to store the encoded data or send it to a network stream, thereby achieving screen recording or screen sharing.

[0003] After obtaining the image data to be encoded, the traditional technology generally transmits the entire frame of the image data to the encoder for encoding. This method requires higher bandwidth resources and affects the data transmission efficiency. Summary of the invention

[0004] To this end, this application discloses the following technical solutions:

[0005] An image processing method, comprising:

[0006] Get the current image frame to be processed;

[0007] Determining a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame;

[0008] In response to the target type of the image frame to be processed being the second type, determining difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed;

[0009] The target information including the first position information is transmitted to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data.

[0010] Optionally, obtaining the current image frame to be processed includes:

[0011] An image frame corresponding to current display data of a display device of the electronic device is obtained from a display memory of a display card of the electronic device as the image frame to be processed.

[0012] Optionally, determining the target type of the image frame to be processed includes:

[0013] Dividing the image frames to be processed into corresponding image picture groups based on a picture grouping strategy; the image picture groups include a plurality of continuous image frames;

[0014] If the image frame to be processed is the first image frame in the image picture group, determining that the target type of the image frame to be processed is the first type;

[0015] If the image frame to be processed is not the first image frame in the image picture group, it is determined that the target type of the image frame to be processed is the second type.

[0016] Optionally, the determining of difference data between the image frame to be processed and the corresponding target reference frame and first position information of a position of the difference data in the image frame to be processed includes:

[0017] Determine a difference pixel block or a difference pixel point between the image frame to be processed and the target reference frame; the difference pixel block includes a plurality of continuous pixel points, and the difference data includes a pixel value corresponding to the difference pixel block or the difference pixel point;

[0018] Determine a first identifier of the image frame to be processed and position information of the difference pixel block or the difference pixel point in the image frame to be processed; the first position information includes the first identifier and position information of the difference pixel block or the difference pixel point in the image frame to be processed.

[0019] Optionally, the image encoding device is an image encoding module on a graphics card of an electronic device;

[0020] After obtaining the current image frame to be processed, the method further includes: storing the image frame to be processed in a memory of the electronic device;

[0021] In the case where only the first position information is transmitted to the image encoding device, the image encoding device obtains the difference data based on the target information, including:

[0022] The image encoding module in the graphics card acquires the difference data corresponding to the first position information from the graphics card memory according to the first position information and the data mapping relationship between the memory of the electronic device and the graphics card memory;

[0023] The data mapping relationship includes a mapping relationship between corresponding storage locations of data in the memory of the electronic device and the video memory of the graphics card.

[0024] Optionally, generating an encoding result corresponding to the image frame to be processed based on the difference data includes:

[0025] Obtaining the preprocessed difference data and the preprocessed target reference frame;

[0026] Integrate the preprocessed difference data into the position corresponding to the first position information in the preprocessed target reference frame to obtain the preprocessed image frame to be processed;

[0027] The preprocessed image frame to be processed is encoded to obtain an encoding result corresponding to the image frame to be processed.

[0028] Optionally, the preprocessed difference data includes a conversion result obtained by performing a color space conversion on the difference data; and the preprocessed target reference frame includes a conversion result obtained by performing a color space conversion on the target reference frame.

[0029] Optionally, the image processing method further includes:

[0030] In response to the target type of the image frame to be processed being the first type, second position information corresponding to the image frame to be processed is transmitted to the image encoding device, so that the image encoding device obtains the image frame to be processed based on the second position information and generates an encoding result corresponding to the image frame to be processed.

[0031] An image encoding device, comprising:

[0032] An acquisition unit, used for acquiring a current image frame to be processed;

[0033] A first determining unit is used to determine a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame;

[0034] A second determining unit, configured to determine, in response to the target type of the image frame to be processed being the second type, difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed;

[0035] A transmission unit is used to transmit target information including the first position information to an image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data.

[0036] An electronic device, comprising:

[0037] A memory for storing at least one set of computer instructions;

[0038] An image coding device, used for image coding processing;

[0039] The processor is configured to implement the following processing steps by executing the instruction set stored in the memory:

[0040] Get the current image frame to be processed;

[0041] Determining a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame;

[0042] In response to the target type of the image frame to be processed being the second type, determining difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed;

[0043] The target information including the first position information is transmitted to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 It is a flowchart of the image processing method provided by this application;

[0046] Figure 2 It is the implementation logic diagram of data transmission and image encoding provided by this application;

[0047] Figure 3 It is a schematic diagram of a flow chart of the image encoding device provided by the present application generating an encoding result corresponding to an image frame to be processed;

[0048] Figure 4 is another flow chart of the image processing method provided by the present application;

[0049] Figure 5 It is a structural diagram of the image processing device provided by the present application;

[0050] Figure 6 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The embodiments of the present application provide an image processing method, device and electronic device for solving the problem that in the conventional technology, when transmitting the image data to be encoded to an encoder for encoding, the bandwidth resource requirement is high and the data transmission efficiency is affected.

[0053] The image processing method provided in the embodiments of the present application can be applied to electronic devices in many general or special computing device environments or configurations, such as personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, etc.

[0054] See also Figure 1 As shown in the method flow chart, the image processing method provided in the embodiment of the present application may at least include the following steps 101 to 104, and these steps are described in detail below.

[0055] Step 101: Obtain the current image frame to be processed.

[0056] Among them, an image frame currently to be encoded in the encoding process of an image or video can be obtained as the image frame to be processed.

[0057] The method provided in the embodiment of the present application can be applied to, but not limited to, application scenarios such as screen sharing, screen projection, and screen recording, for example, a scenario in which a first electronic device shares or projects a screen to a second electronic device, or a scenario in which a screen is recorded on an electronic device to store the recorded screen data, etc. For these application scenarios, an image frame corresponding to the current display data of the electronic device display device can be obtained from the video memory of the electronic device graphics card as the image frame to be processed.

[0058] Among them, for the scenario where the first electronic device performs screen sharing or screen projection to the second electronic device, the first electronic device can obtain the image frame corresponding to the current display content on its display screen, and use the obtained image frame as the current image frame to be processed to perform the required processing to achieve encoding, and then after completing the encoding of the image frame to be processed, send the obtained encoded data to the second electronic device at the sharing destination or the screen projection destination (such as sending the network stream / data stream of the encoded data to the second electronic device through the network interface), so as to achieve screen sharing or screen projection to the second electronic device. In this application scenario, optionally, the CPU (Central Processing Unit, central processing unit) or other processing devices of the first electronic device can obtain the image frame corresponding to the current display content on its display screen from the video memory of the graphics card, and use the obtained image frame as the current image frame to be processed.

[0059] Similarly, for the screen recording scenario, the electronic device that is recording the screen can obtain the image frame corresponding to the current display content on its display screen, and use the obtained image frame as the current image frame to be processed to perform the required processing to achieve encoding, and then store the obtained encoded data after completing the encoding of the image frame to be processed to achieve screen recording. In this application scenario, optionally, a processing device such as a CPU of the electronic device that is recording the screen can obtain the image frame corresponding to the current display content on its display screen from the video memory of the graphics card of the electronic device, and use the obtained image frame as the current image frame to be processed.

[0060] After the CPU or other processing device of the electronic device obtains the image frame corresponding to the current display content on its display screen (i.e., the image frame to be processed) from the video memory of the graphics card, the obtained image frame can be optionally stored in the memory of the electronic device for subsequent processing.

[0061] Step 102 : determining a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame.

[0062] Optionally, the image frame to be processed can be first divided into corresponding image picture groups based on the picture grouping strategy, and then it is further determined whether the image frame to be processed is the first image frame (i.e., the first frame) in the image picture group to which it belongs, wherein, if the image frame to be processed is the first image frame in the image picture group to which it belongs, the target type of the image frame to be processed is determined to be the first type; conversely, if the image frame to be processed is not the first image frame in the image picture group to which it belongs, the target type of the image frame to be processed is determined to be the second type.

[0063] The image picture group includes a plurality of consecutive image frames.

[0064] The picture grouping strategy may be, but is not limited to, a picture grouping strategy based on GOP (Group of Pictures). In the picture grouping strategy of the embodiment of the present application, the size of the picture group is set, and the size of the picture group represents the maximum number of image frames that each picture group can contain. Assuming that the size of the picture group is set to N, it means that when grouping the image frames, a continuous number of N image frames needs to be divided into one picture group. The N is an integer greater than 1.

[0065] Exemplarily, in the case where a processing device such as a CPU obtains the current image frame to be processed from the video memory of a graphics card and stores it into the internal memory, the CPU can specifically divide the image frame to be processed into a corresponding image screen group based on a screen grouping strategy, and determine the target type of the image frame to be processed based on a determination result of whether the image frame to be processed is the first image frame in the image screen group to which it belongs.

[0066] Step 103 : in response to the target type of the image frame to be processed being the second type, determining difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position of the difference data in the image frame to be processed.

[0067] The target type of the image frame to be processed is the second type, indicating that the image frame to be processed is not the first image frame in the image picture group to which it belongs, and correspondingly indicating that the image frame to be processed is not a reference frame in the image picture group to which it belongs.

[0068] The target reference frame corresponding to the image frame to be processed refers to the first image frame in the image picture group to which the image frame to be processed belongs.

[0069] If the target type of the image frame to be processed is the second type, this embodiment determines difference data between the image frame to be processed and its corresponding target reference frame, and determines first position information of the position of the difference data in the image frame to be processed.

[0070] Optionally, a difference pixel block or difference pixel point between the image frame to be processed and the target reference frame may be specifically determined, and the pixel value corresponding to the difference pixel block or difference pixel point may be used as the difference data between the image frame to be processed and its corresponding target reference frame.

[0071] The difference pixel points between the image frame to be processed and the target reference frame specifically refer to the pixel points whose pixel values ​​in the image frame to be processed are different from the pixel values ​​at the corresponding positions in the target reference frame.

[0072] The difference pixel block includes a plurality of continuous pixels. Among the plurality of continuous pixels included in the difference pixel block, there are at least a plurality of difference pixel points that meet a neighboring condition. The neighboring condition may be, but is not limited to, set as a distance between pixel positions corresponding to different difference pixel points being less than a set threshold.

[0073] That is, the difference data may include pixel values ​​corresponding to each difference pixel point between the image frame to be processed and its corresponding target reference frame, wherein the positions corresponding to each difference pixel point in the image frame to be processed may be continuous or discontinuous. Alternatively, the difference data may also include pixel values ​​corresponding to difference pixel blocks between the image frame to be processed and its corresponding target reference frame, wherein the number of the difference pixel blocks may be one or more, and the positions corresponding to different difference pixel blocks in the image frame to be processed may be continuous or discontinuous, depending on the actual application.

[0074] The pixel values ​​corresponding to the difference pixel block include the pixel values ​​corresponding to each pixel point in the difference pixel block.

[0075] When determining the first position information of the position of the difference data in the image frame to be processed, the first identifier of the image frame to be processed and the position information of the difference pixel block or the difference pixel point in the image frame to be processed can be specifically determined, and the first identifier of the image frame to be processed and the position information of the difference pixel block / difference pixel point in the image frame to be processed are used as the first position information, that is, the first position information includes: the first identifier of the image frame to be processed and the position information of the difference pixel block / difference pixel point in the image frame to be processed.

[0076] The first identifier of the image frame to be processed may specifically be, but is not limited to, a frame sequence number or a frame number corresponding to the image frame to be processed.

[0077] Among them, in the case where the difference data includes pixel values ​​of each difference pixel point between the image frame to be processed and its corresponding target reference frame, the pixel coordinates corresponding to each difference pixel point in the image frame to be processed can be specifically determined as the position information of each difference pixel point in the image frame to be processed.

[0078] In the case where the difference data includes pixel values ​​of a difference pixel block between the image frame to be processed and its corresponding target reference frame, optionally, the difference pixel block can be a rectangular pixel block formed by multiple continuous pixel points, such as a 16*16 rectangular pixel block, etc. For this case, the pixel point positions on preset vertices of the difference pixel block (such as the pixel coordinates of the upper left corner vertex of the rectangular pixel block) can be specifically determined, and the size information such as the width and height of the difference pixel block can be determined, and the pixel point positions on the preset vertices and the size information such as the width and height are used as the position information of the difference pixel block in the image frame to be processed.

[0079] Step 104 : Transmit the target information including the first position information to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates the encoding result corresponding to the image frame to be processed based on the difference data.

[0080] Optionally, the image encoding device may be an image encoding module on a graphics card of the electronic device.

[0081] When transmitting the target information including the first position information to the image encoding device, an optional implementation is to transmit only the first position information to the image encoding device, in which the target information is the first position information. Wherein, in the case where the difference data includes the pixel values ​​of each difference pixel point between the image frame to be processed and its corresponding target reference frame, specifically only the first identifier of the image frame to be processed and the pixel coordinates of each difference pixel point can be transmitted to the image encoding device; in the case where the difference data includes the pixel values ​​of the difference pixel block between the image frame to be processed and its corresponding target reference frame, only the first identifier of the image frame to be processed, the pixel point position on the preset vertex of the difference pixel block (such as the pixel coordinates of the upper left vertex) and the size information such as the width and height of the difference pixel block can be transmitted to the image encoding device.

[0082] In this implementation, an image encoding device, such as an image encoding module on the graphics card side of an electronic device, can specifically obtain the difference data matching the first position information, such as the pixel values ​​of the individual difference pixels or the pixel values ​​of the difference pixel blocks, from the graphics card's video memory based on the received first position information, and determine the complete image data of the image frame to be processed based on the image data of the target reference frame, the difference data and the first position information, and then encode the complete image data of the image frame to be processed to generate an encoding result corresponding to the image frame to be processed.

[0083] In practical applications, it is not limited to the above implementations. In other implementations, this step may also transmit the difference data and the first position information to the image encoding device. In this implementation, the target information includes the difference data and the first position information.

[0084] In this embodiment, an image encoding device, such as an image encoding module on the graphics card side of an electronic device, can directly determine the complete image data of the image frame to be processed based on the image data of the target reference frame, the difference data and the first position information, and then encode the complete image data of the image frame to be processed to generate an encoding result corresponding to the image frame to be processed.

[0085] It is worth noting that the image encoding device has obtained the image data of the target reference frame when encoding the target reference frame. Since the target reference frame is the first image frame in the image picture group to which the image frame to be processed belongs, the encoding time of the target reference frame is earlier than the encoding time of the image frame to be processed, and the image encoding device will correspondingly obtain the image data of the target reference frame earlier than the image frame to be processed.

[0086] In summary, the image processing method provided by the embodiment of the present application determines the target type of the image frame to be processed after obtaining the current image frame to be processed, and in response to the target type of the image frame to be processed being the second type characterizing that the image frame to be processed is a non-reference frame, determines the difference data between the image frame to be processed and the corresponding target reference frame and the first position information of the position of the difference data in the image frame to be processed, and then transmits the target information containing the first position information to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates the encoding result corresponding to the image frame to be processed based on the difference data. It can be seen that for the image frame to be processed, when it is a non-reference frame, the present application can avoid transmitting at least part of the image data of the image frame to be processed to the image encoding device through the above processing while enabling the image encoding device to normally encode the image frame to be processed, and the data amount corresponding to the first position information is small, which is much smaller than the data amount of the image data exempted from transmission, so that the present application reduces the data transmission amount and bandwidth demand, and accordingly achieves the effect of saving bandwidth demand and improving data transmission efficiency.

[0087] In an optional embodiment, the image encoding device is an image encoding module on a graphics card side of the electronic device, such as an encoder on a graphics card side.

[0088] In this embodiment, step 104 of the method of the present application only transmits the first position information to the image encoding device.

[0089] In the case where only the first position information is transmitted to the image encoding device, the image encoding device obtains the difference data based on the target information, which can be implemented as follows: the image encoding module in the graphics card obtains the difference data corresponding to the first position information from the graphics card's memory according to the first position information and the data mapping relationship between the memory of the electronic device and the graphics card's memory.

[0090] In the present application examples, see Figure 2 As shown in the implementation logic diagram of data transmission and image encoding, there is a data mapping relationship between the memory of the electronic device and the video memory of the graphics card, for example, there is a data mapping relationship between the memory of the electronic device and the Display Vram (Video RAM, image random access memory) of the graphics card, and the data mapping relationship includes the mapping relationship between the corresponding storage locations of the data in the memory of the electronic device and the video memory (such as Display Vram) of the graphics card.

[0091] The image encoding module in the graphics card can specifically determine the target storage location of the image frame to be processed in the graphics card memory based on the data mapping relationship and the first identifier of the image frame to be processed in the first position information, such as the frame number of the image frame to be processed, and then extract the difference data from the image frame to be processed stored in the target storage location according to the position information of the position of the difference data in the image frame to be processed in the first position information.

[0092] For example, according to the pixel coordinates of each difference pixel point in the image frame to be processed contained in the first position information, the pixel value of each difference pixel point is extracted from the image frame to be processed stored in the target storage position, or, according to the corresponding position information of the difference pixel block in the image frame to be processed in the first position information (such as the upper left corner pixel coordinates of the rectangular pixel block and the width and height of the rectangular pixel block), the pixel value of the difference pixel block is extracted from the image frame to be processed stored in the target storage position.

[0093] On the basis of obtaining the difference data, subsequently, the complete image data of the image frame to be processed can be determined according to the image data of the target reference frame, the difference data and the first position information, and then the complete image data of the image frame to be processed can be encoded to generate an encoding result corresponding to the image frame to be processed. This part will be described in detail in the following embodiments.

[0094] For the image frame to be processed, when it is a non-reference frame, this embodiment only transmits the first position information to the image encoding device, and the image encoding device obtains the difference data from the video memory of the graphics card based on the first position information (the image encoding device is an image encoding module on the graphics card side, and obtaining the image data from the video memory of the graphics card does not consume bandwidth). This enables the image encoding device to normally encode the non-reference frame while avoiding transmitting the image data of the non-reference frame to the image encoding device, thereby greatly reducing the transmission amount of image data and the bandwidth requirement, and correspondingly achieving the effect of saving bandwidth requirements and improving data transmission efficiency.

[0095] In an alternative embodiment, see Figure 3 As shown in the flowchart, the image encoding device generates the encoding result corresponding to the image frame to be processed based on the difference data, which can be specifically implemented as follows:

[0096] Step 301: Obtain the preprocessed difference data and the preprocessed target reference frame.

[0097] Optionally, the preprocessing includes color space conversion processing on the image data, so as to convert the image data into a data format required for image encoding by the image encoding device based on the color space conversion processing.

[0098] The preprocessed difference data may include a conversion result obtained by performing a color space conversion on the difference data; the preprocessed target reference frame may include a conversion result obtained by performing a color space conversion on the target reference frame.

[0099] The color space conversion process may include, but is not limited to, conversion from ARGB to YUV color space. ARGB refers to the RGB (Red-Green-Blue) color mode with an Alpha (transparency) channel added; YUV is a color encoding method, "Y" represents brightness (Luminance or Luma), that is, the grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0100] Optionally, a GPU (Graphics Processing Unit) in a graphics card may be used to perform a color space conversion process from ARGB to YUV on the difference data or the target reference frame.

[0101] The preprocessed target reference frame (such as the target reference frame converted into the YUV format) is the result obtained by preprocessing the target reference frame before performing encoding processing on the target reference frame. After obtaining the preprocessed target reference frame, it can be cached to the image encoding device end, so that when each non-reference frame in the image picture group to which the target reference frame belongs is subsequently encoded, the cached preprocessed target reference frame can be reused as a base image.

[0102] Among them, Figure 2 As shown, the base map, that is, the reference frame in each group of image frames, can be refreshed at regular intervals as the image frames involved in the encoding are replaced.

[0103] Therefore, in this step, the image encoding device can specifically obtain the pre-processed target reference frame by reading the cache.

[0104] For the difference data, after extracting the difference data from the video memory of the graphics card according to the first position information, or obtaining the difference data transmitted by the CPU, the image encoding device can transmit the difference data to the GPU, and the GPU performs color space conversion processing from ARGB to YUV, etc. on the difference data to complete preprocessing of the difference data and obtain the preprocessed difference data, such as obtaining the pixel value of each difference pixel point after preprocessing, or the pixel value of each difference pixel block after preprocessing, etc.

[0105] Among them, the pixel value of each difference pixel point after preprocessing may specifically include the YUV format pixel value corresponding to each difference pixel point after conversion from ARGB to YUV; similarly, the pixel value of each difference pixel block after preprocessing may include the YUV format pixel value corresponding to the pixel point in the difference pixel block after conversion from ARGB to YUV.

[0106] Step 302: Integrate the preprocessed difference data into the position corresponding to the first position information in the preprocessed target reference frame to obtain the preprocessed image frame to be processed.

[0107] After obtaining the preprocessed difference data, refer to Figure 2 , the preprocessed difference data can be further integrated / compensated to the corresponding position in the preprocessed target reference frame according to the position information corresponding to the difference data in the first position information in the image to be processed, so as to obtain the preprocessed image frame to be processed, such as obtaining the image frame to be processed in YUV format.

[0108] Step 303: Encode the preprocessed image frame to be processed to obtain an encoding result corresponding to the image frame to be processed.

[0109] The preprocessed image frame to be processed, such as the image frame to be processed in YUV format, meets the format requirements of the image encoding device when encoding image data. Therefore, after obtaining the preprocessed image frame to be processed, the image encoding device can be used to encode it and generate an encoding result corresponding to the output of the image frame to be processed.

[0110] After obtaining the encoding result (i.e., the encoded data) corresponding to the image frame to be processed, the required processing can be performed on the encoded data of the image frame to be processed according to the needs of the actual application scenario. For example, in the screen sharing / screen projection application scenario, the encoded data of the image frame to be processed is sent to the second electronic device at the screen sharing / screen projection destination, thereby realizing screen sharing or screen projection to the second electronic device. In the screen recording scenario, the encoded data of the image frame to be processed can be stored to realize screen recording.

[0111] For the case where the image frame to be processed is a non-reference frame, since the embodiment of the present application only transmits the first position information corresponding to the difference data of the image frame to be processed compared with the target reference frame to the image encoding device, or only transmits the difference data and the first position information, it avoids transmitting at least part of the image data of the image frame to be processed to the image encoding device (the amount of data corresponding to the first position information is small, much smaller than the amount of image data exempted from transmission), which correspondingly reduces the data transmission amount and bandwidth requirement, and can achieve the effect of saving bandwidth requirements and improving data transmission efficiency. At the same time, it can also ensure the encoding efficiency of the image encoding device and avoid affecting the image encoding efficiency due to too long data transmission time.

[0112] In an alternative embodiment, see Figure 4 The image processing method flow chart shown in the figure, the image processing method provided by the embodiment of the present application may also include the following processing:

[0113] Step 105, in response to the target type of the image frame to be processed being the first type, transmitting second position information corresponding to the image frame to be processed to the image encoding device, so that the image encoding device obtains the image frame to be processed based on the second position information and generates an encoding result corresponding to the image frame to be processed.

[0114] The target type of the image frame to be processed is the first type, which indicates that the image frame to be processed is a reference frame, that is, the first image frame in the image picture group to which it belongs.

[0115] The second position information corresponding to the image frame to be processed may include the frame sequence number or frame number of the image frame to be processed. In addition, it may optionally include pixel position information of the image frame to be processed (such as the pixel coordinates of the upper left corner pixel of the image frame to be processed, and the width and height of the image frame to be processed, etc.).

[0116] When the target type of the image frame to be processed is the first type, this embodiment only transmits the second position information corresponding to the image frame to be processed to the image encoding device. Optionally, the CPU can transmit the second position information corresponding to the image frame to be processed to an image encoding device such as an encoder on the graphics card end. After the image encoding device receives the second position information corresponding to the image frame to be processed, it obtains the image frame to be processed corresponding to the second position information from the graphics card's memory according to the second position information and the data mapping relationship between the internal memory and the graphics card's memory.

[0117] Afterwards, the acquired image frame to be processed may be transmitted to the GPU, and the GPU may perform a color space conversion process from ARGB to YUV on the image frame to be processed, so as to complete the preprocessing of the image frame to be processed.

[0118] After completing the preprocessing of the image frame to be processed and obtaining the preprocessed image frame to be processed, such as the image frame to be processed in YUV format, the image encoding device can be used to encode the preprocessed image frame to be processed to obtain the encoding result corresponding to the image frame to be processed. Afterwards, the encoded data of the image frame to be processed can be further processed according to the needs of the actual application scenario. For example, in a screen sharing / screen projection application, the encoded data of the image frame to be processed is sent to the second electronic device at the screen sharing / screen projection destination, thereby realizing screen sharing or screen projection to the second electronic device. In a screen recording scenario, the encoded data of the image frame to be processed can be stored to realize screen recording.

[0119] In the case where the image frame to be processed is a reference frame, the preprocessed image frame to be processed, such as a reference frame in YUV format, is also cached to the image encoding device end so as to be subsequently multiplexed as a base image for each non-reference frame in the image picture group to which the reference frame belongs.

[0120] In this embodiment, in the case where the image frame to be processed is a reference frame, by only transmitting the second position information corresponding to the image frame to be processed to the image encoding device, and the image encoding device obtains the image frame to be processed from the graphics card memory based on the second position information (the image encoding device is an image encoding module on the graphics card side, and obtaining image data from the graphics card memory does not consume bandwidth), while enabling the image encoding device to encode the reference frame normally, it can also avoid transmitting the image data of the reference frame to the image encoding device, thereby further reducing the data transmission amount and bandwidth demand, and correspondingly achieving the effect of further saving bandwidth demand and improving data transmission efficiency. In addition, it can also reduce the load of the graphics card, GPU / CPU, etc., and improve the encoding frame rate and efficiency.

[0121] In practical applications, when the image frame to be processed is a reference frame, the image data of the reference frame can also be directly transmitted to the image encoding device. Since for the non-reference frame, at least part of its image data is avoided from being transmitted to the image encoding device, compared with the traditional technology of transmitting the entire frame of image data to be encoded to the encoder for encoding, this embodiment can still achieve a series of technical effects such as reducing the amount of data transmission, reducing bandwidth requirements, improving data transmission efficiency, reducing the load on the graphics card / GPU / CPU, etc., and improving the encoding frame rate and efficiency.

[0122] Corresponding to the above-mentioned image processing method, the embodiment of the present application further provides an image processing device, whose composition structure is as follows: Figure 5 As shown, including:

[0123] An acquisition unit 501 is used to obtain a current image frame to be processed;

[0124] A first determining unit 502 is used to determine a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame;

[0125] A second determining unit 503 is configured to determine, in response to the target type of the image frame to be processed being the second type, difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed;

[0126] The transmission unit 504 is used to transmit the target information including the first position information to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates the encoding result corresponding to the image frame to be processed based on the difference data.

[0127] In an optional implementation manner, the acquisition unit 501 is specifically configured to:

[0128] An image frame corresponding to current display data of a display device of the electronic device is obtained from a display memory of a display card of the electronic device as the image frame to be processed.

[0129] In an optional implementation manner, the first determining unit 502 is specifically configured to:

[0130] Dividing the image frames to be processed into corresponding image picture groups based on a picture grouping strategy; the image picture groups include a plurality of continuous image frames;

[0131] If the image frame to be processed is the first image frame in the image picture group, determining that the target type of the image frame to be processed is the first type;

[0132] If the image frame to be processed is not the first image frame in the image picture group, it is determined that the target type of the image frame to be processed is the second type.

[0133] In an optional implementation manner, the second determination unit 503, when determining the difference data between the image frame to be processed and the corresponding target reference frame and the first position information of the position where the difference data is located in the image frame to be processed, is specifically used to:

[0134] Determine a difference pixel block or a difference pixel point between the image frame to be processed and the target reference frame; the difference pixel block includes a plurality of continuous pixel points, and the difference data includes a pixel value corresponding to the difference pixel block or the difference pixel point;

[0135] Determine a first identifier of the image frame to be processed and position information of the difference pixel block or the difference pixel point in the image frame to be processed; the first position information includes the first identifier and position information of the difference pixel block or the difference pixel point in the image frame to be processed.

[0136] In an optional implementation manner, the image encoding device is an image encoding module on a graphics card of an electronic device;

[0137] The device also includes a storage processing module, which is used to store the image frame to be processed in the memory of the electronic device after obtaining the current image frame to be processed;

[0138] In the case where only the first position information is transmitted to the image encoding device, the image encoding device obtains the difference data based on the target information, including:

[0139] The image encoding module in the graphics card acquires the difference data corresponding to the first position information from the graphics card memory according to the first position information and the data mapping relationship between the memory of the electronic device and the graphics card memory;

[0140] The data mapping relationship includes a mapping relationship between corresponding storage locations of data in the memory of the electronic device and the video memory of the graphics card.

[0141] In an optional implementation manner, the image encoding device, when generating the encoding result corresponding to the image frame to be processed based on the difference data, is specifically configured to:

[0142] Obtaining the preprocessed difference data and the preprocessed target reference frame;

[0143] Integrate the preprocessed difference data into the position corresponding to the first position information in the preprocessed target reference frame to obtain the preprocessed image frame to be processed;

[0144] The preprocessed image frame to be processed is encoded to obtain an encoding result corresponding to the image frame to be processed.

[0145] In an optional implementation, the preprocessed difference data includes a conversion result obtained by performing a color space conversion on the difference data; and the preprocessed target reference frame includes a conversion result obtained by performing a color space conversion on the target reference frame.

[0146] In an optional embodiment, the transmission unit 504 is further used to: in response to the target type of the image frame to be processed being the first type, transmit second position information corresponding to the image frame to be processed to the image encoding device, so that the image encoding device obtains the image frame to be processed based on the second position information, and generates an encoding result corresponding to the image frame to be processed.

[0147] The present application also provides an electronic device, the composition structure of the electronic device is as follows: Figure 6 As shown, including at least:

[0148] A memory 10, for storing a computer instruction set;

[0149] A set of computer instructions can be implemented in the form of a computer program.

[0150] An image coding device 20, used for image coding processing;

[0151] The processor 30 is used to implement the image processing method provided by any of the above method embodiments by executing a computer instruction set.

[0152] The processor 30 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a neural network processor (NPU), a deep learning processor (DPU) or other programmable logic devices, etc.

[0153] The electronic device includes a display device and / or includes a display interface and can be connected to an external display device.

[0154] Optionally, the electronic device may further include a graphics card, a GPU.

[0155] In addition, the electronic device may also include components such as a communication interface and a communication bus. The memory, the processor and the communication interface communicate with each other through the communication bus.

[0156] The communication interface is used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0157] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0158] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0159] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or the part that makes a creative contribution can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0160] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0161] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, comprising: Get the current image frame to be processed; Determining the target type of the image frame to be processed; The target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame; In response to the target type of the image frame to be processed being the second type, determining difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed; The target information including the first position information is transmitted to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data.

2. The image processing method according to claim 1, wherein obtaining the current image frame to be processed comprises: An image frame corresponding to current display data of a display device of the electronic device is obtained from a display memory of a display card of the electronic device as the image frame to be processed.

3. The image processing method according to claim 1, wherein determining the target type of the image frame to be processed comprises: Dividing the image frames to be processed into corresponding image picture groups based on a picture grouping strategy; The image picture group includes a plurality of continuous image frames; If the image frame to be processed is the first image frame in the image picture group, determining that the target type of the image frame to be processed is the first type; If the image frame to be processed is not the first image frame in the image picture group, it is determined that the target type of the image frame to be processed is the second type.

4. The image processing method according to claim 1, wherein the step of determining the difference data between the image frame to be processed and the corresponding target reference frame and the first position information of the position of the difference data in the image frame to be processed comprises: Determine a difference pixel block or a difference pixel point between the image frame to be processed and the target reference frame; The difference pixel block includes a plurality of continuous pixel points, and the difference data includes pixel values ​​corresponding to the difference pixel block or the difference pixel points; Determine a first identifier of the image frame to be processed and position information of the difference pixel block or the difference pixel point in the image frame to be processed; the first position information includes the first identifier and position information of the difference pixel block or the difference pixel point in the image frame to be processed.

5. The image processing method according to claim 1, wherein the image encoding device is an image encoding module on a graphics card of an electronic device; After obtaining the current image frame to be processed, the method further includes: storing the image frame to be processed in a memory of the electronic device; In the case where only the first position information is transmitted to the image encoding device, the image encoding device obtains the difference data based on the target information, including: The image encoding module in the graphics card acquires the difference data corresponding to the first position information from the graphics card memory according to the first position information and the data mapping relationship between the memory of the electronic device and the graphics card memory; The data mapping relationship includes a mapping relationship between corresponding storage locations of data in the memory of the electronic device and the video memory of the graphics card.

6. The image processing method according to claim 1, wherein generating the encoding result corresponding to the image frame to be processed based on the difference data comprises: Obtaining the preprocessed difference data and the preprocessed target reference frame; Integrate the preprocessed difference data into the position corresponding to the first position information in the preprocessed target reference frame to obtain the preprocessed image frame to be processed; The preprocessed image frame to be processed is encoded to obtain an encoding result corresponding to the image frame to be processed.

7. According to the image processing method of claim 6, the pre-processed difference data includes a conversion result obtained by performing a color space conversion on the difference data; the pre-processed target reference frame includes a conversion result obtained by performing a color space conversion on the target reference frame.

8. The image processing method according to claim 1, further comprising: In response to the target type of the image frame to be processed being the first type, second position information corresponding to the image frame to be processed is transmitted to the image encoding device, so that the image encoding device obtains the image frame to be processed based on the second position information and generates an encoding result corresponding to the image frame to be processed.

9. An image encoding device, comprising: An acquisition unit, used for acquiring a current image frame to be processed; A first determining unit, configured to determine a target type of the image frame to be processed; The target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame; A second determining unit, configured to determine, in response to the target type of the image frame to be processed being the second type, difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed; A transmission unit is used to transmit target information including the first position information to an image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data.

10. An electronic device, comprising: A memory for storing at least one set of computer instructions; An image coding device, used for image coding processing; The processor is configured to implement the following processing steps by executing the instruction set stored in the memory: Get the current image frame to be processed; Determining a target type of the image frame to be processed; the target type of the image frame is a first type representing that the image frame is a reference frame, or a second type representing that the image frame is a non-reference frame; In response to the target type of the image frame to be processed being the second type, determining difference data between the image frame to be processed and a corresponding target reference frame and first position information of a position where the difference data is located in the image frame to be processed; The target information including the first position information is transmitted to the image encoding device, so that the image encoding device obtains the difference data based on the target information, and generates an encoding result corresponding to the image frame to be processed based on the difference data.

Images