Image processing method, apparatus and electronic device

Abstract

The application discloses an image processing method, device and electronic equipment. After obtaining a to-be-processed image, the method, device and electronic equipment perform preprocessing on the to-be-processed image to obtain preprocessing result information. The preprocessing at least includes processing represented by common processing logic of at least part of image processing units in an image processing unit set. Then, the to-be-processed image and the preprocessing result information are output to the image processing unit set, so that the image processing unit set performs processing logic other than the preprocessing according to the preprocessing result information on the to-be-processed image.

Description

Technical Field

[0001] This application relates to an image processing method, apparatus, and electronic device. Background Technology

[0002] The video capture stack in Windows is a camera device extension component that supports user-mode extensions using DMFT (Device Media Foundation Transform). DMFT can receive raw image frames from the camera device and perform post-processing operations on the image frames internally. See also Figure 1 The DMFT structure shown can contain multiple MFT (Media Foundation Transform) image processing plugins, which perform different image processing tasks on the image frames.

[0003] Since each raw image frame output by the camera device needs to be processed by the MFT chain (MFTs) in the DMFT, the total processing time for each image frame is relatively long, which may eventually cause the camera preview image to stutter or drop frames. Summary of the Invention

[0004] Therefore, this application discloses the following technical solution:

[0005] An image processing method, the method comprising:

[0006] Obtain the image to be processed;

[0007] Preprocessing is performed on the image to be processed to obtain preprocessing result information; the preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set;

[0008] The image to be processed and the preprocessing result information are output to the image processing unit set, so that the image processing unit set performs processing logic other than preprocessing in the corresponding processing logic of the image to be processed according to the preprocessing result information.

[0009] Optionally, the image processing units included in the image processing unit set are connected in a chain-like manner to form an image processing chain;

[0010] The step of outputting the image to be processed and the preprocessing result information to the image processing unit set includes:

[0011] The image to be processed and the preprocessing result information are output to the image processing unit at the head of the image processing chain.

[0012] Optionally, outputting the image to be processed and the preprocessing result information to the image processing unit set includes:

[0013] The image to be processed and the preprocessing result information are fused to obtain a first fused image, which is then output to the image processing unit set.

[0014] Optionally, the image processing unit set performs processing logic other than the preprocessing in the corresponding processing logic on the image to be processed based on the preprocessing result information, including:

[0015] The image processing unit at the head of the image processing chain is taken as the current image processing unit;

[0016] The current image processing unit uses the corresponding information in the preprocessing result information to perform processing logic other than preprocessing on the received image in the corresponding processing logic, and uses the processed image as the current intermediate image;

[0017] The current image processing unit outputs the preprocessing result information and the intermediate image it generates to the next image processing unit in the image processing chain.

[0018] The current image processing unit is updated based on the next image processing unit, and the processing procedure for the current image processing unit is executed cyclically until the image generated by the image processing unit at the end of the image processing chain is obtained. Then, the image generated by the image processing unit at the end of the image processing chain is used as the processing result image of the image to be processed.

[0019] Optionally, before outputting the preprocessing result information and the intermediate image generated by the current image processing unit to the next image processing unit in the image processing chain, the method further includes:

[0020] The current image processing unit performs a fusion process on the preprocessing result information and the intermediate image it generates to obtain a second fused image, which is then output to the next image processing unit in the image processing chain.

[0021] Optionally, the corresponding image and the preprocessing result information are fused to obtain a fused image, including:

[0022] Obtain a notification message in a preset format obtained by encapsulating the preprocessed result information;

[0023] Using preset bits of multiple different bytes in the byte information of the corresponding image, different bit information of the notification message is stored sequentially to obtain a fused image of the corresponding image and the preprocessing result information;

[0024] The corresponding image is either the image to be processed or the corresponding intermediate image.

[0025] Optionally, the preset bit is the least significant bit of the byte.

[0026] Optionally, before the current image processing unit performs processing logic other than preprocessing on the received image according to the corresponding information in the preprocessing result information, the method further includes:

[0027] The current image processing unit extracts preset bit information of each byte of the received fused image, and concatenates the extracted information into a corresponding string in sequence;

[0028] The notification message is extracted from the string according to the set extraction rules.

[0029] An image processing apparatus, the apparatus comprising:

[0030] A preprocessing unit is used to acquire an image to be processed, perform preprocessing on the image to be processed, obtain preprocessing result information, and output the image to be processed and the preprocessing result information to the image processing unit set;

[0031] An image processing unit set is used to receive the image to be processed and the preprocessing result information output by the preprocessing unit, and to execute processing logic other than the preprocessing in the corresponding processing logic on the image to be processed according to the preprocessing result information.

[0032] The preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set.

[0033] An electronic device, comprising:

[0034] Memory, used to store at least one set of computer instructions;

[0035] A processor is configured to implement the image processing method described in any of the preceding descriptions by invoking and executing the instruction set stored in the memory.

[0036] As can be seen from the above solutions, the image processing method, apparatus, and electronic device disclosed in this application, after obtaining the image to be processed, performs preprocessing on the image to be processed to obtain preprocessing result information. The preprocessing performed includes at least: processing of the common processing logic representation of at least some image processing units in the image processing unit set; then, the image to be processed and the preprocessing result information are output to the image processing unit set, so that the image processing unit set performs processing logic other than the preprocessing in the corresponding processing logic of the image to be processed according to the preprocessing result information. This application extracts the common processing logic of at least some image processing units in the image processing unit set and adds a preprocessing step before the image processing of the image processing unit set. By placing the extracted common processing logic in the preprocessing step for execution, it can avoid the repetitive processing work caused by different units in the image processing unit set executing the common processing logic, thereby effectively reducing the total processing time of each image in the image processing unit set and avoiding or alleviating the problem of stuttering or frame dropping in the camera preview image. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a structural diagram of the DMFT;

[0039] Figure 2 This is a flowchart illustrating the image processing method provided in this application;

[0040] Figure 3 This is a comparison diagram of the display effects of red pixel values ​​233 and 232 provided in this application;

[0041] Figure 4 This is a flowchart of the MFTs chain for processing image streams provided in this application;

[0042] Figure 5 This is a structural diagram of the image processing apparatus provided in this application;

[0043] Figure 6 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] This application discloses an image processing method, apparatus, and electronic device to reduce the processing time of camera images and improve the problem of stuttering or frame dropping in camera preview images. The image processing method of this application can be applied to, but is not limited to, electronic devices in a variety of general-purpose or special-purpose computing device environments or configurations, such as: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0046] See Figure 2 The provided image processing method flowchart shows that the image processing method provided in this application embodiment includes the following processing steps:

[0047] Step 201: Obtain the image to be processed.

[0048] The acquisition of the image to be processed can be achieved by receiving photos or video frames from the device's camera in real time during image acquisition scenarios such as taking photos or recording videos, or by reading images or video frames that require image processing from the device's local storage or downloading them from the network; there are no restrictions.

[0049] In this embodiment of the application, the image processing required for the image to be processed is the image processing provided by each image processing unit in the image processing unit set.

[0050] Optionally, the image processing units included in the image processing unit set can be connected in a chain / serial connection to form an image processing chain. Each image processing unit in the image processing chain is used to perform different image processing tasks, such as background replacement / background blurring, face focusing, beautification, soft light, etc. By having different image processing units in the image processing chain perform different image processing tasks on the image, the multi-faceted processing needs of the image can be met.

[0051] Optionally, a set of linked image processing units, i.e. an image processing chain, can be specifically an image processing chain MFTs composed of various MFTs in the DMFT.

[0052] It is easy to understand that the individual image processing units included in the image processing unit set can also be connected into a whole image processing module in other ways, such as by connecting them in a combination of parallel and chain / serial methods, depending on the actual needs.

[0053] Step 202: Perform preprocessing on the image to be processed to obtain preprocessing result information. The preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set.

[0054] The applicant discovered that multiple development teams are involved in the camera MFTs image processing chain to implement different MFT image processing plugins. These MFTs may need to do some common image analysis work. For example, each MFT that needs to process face images will spend time analyzing the face position. This will increase the total processing time of each frame in the MFTs, which may eventually cause the camera preview image to stutter or drop frames.

[0055] Based on this, this application extracts the common processing logic of at least some of the image processing units in the image processing unit set, and adds a preprocessing stage before image processing in the image processing unit set, placing the extracted common processing logic in the preprocessing stage for execution. Accordingly, the preprocessing performed in the preprocessing stage includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set.

[0056] The extracted common processing logic can be the common processing logic of some image processing units in the image processing unit set, or it can be the common processing logic of all image processing units, or it can include both the common processing logic of some image processing units and the common processing logic of all image processing units. There are no restrictions, and it can be determined according to actual needs.

[0057] In addition to the common processing logic that includes at least some of the extracted image processing units, the preprocessing logic executed in the preprocessing stage may optionally include some specific non-common processing logic.

[0058] For example, this specific non-shared processing logic can be some relatively basic processing logic in image processing that is predicted to have a high probability of becoming shared processing logic in different processing units in subsequent functional expansions / improvements. For instance, suppose the human contour analysis logic is not a shared processing logic in some or all MFTs, but after prediction, "human contour analysis" is a relatively basic processing logic in portrait processing, applicable to various image processing scenarios such as background replacement / background blurring, slimming, leg enhancement, and leg stretching. It has a very high probability of becoming a shared processing logic in some or all MFTs after subsequent functional expansions. Therefore, the non-shared processing logic "human contour analysis" can be added to the preprocessing stage along with the extracted shared processing logic, thereby facilitating the expansion and improvement of subsequent image processing functions.

[0059] When applying this application, a preprocessing unit can be added to the existing image processing unit set provided by the device to execute the preprocessing logic of this application. Alternatively, in other embodiments, preprocessing logic can be added to the first image processing unit of the image processing unit set, and the first image processing unit of the image processing unit set is responsible for executing the preprocessing logic. For example, the first image processing unit executes the added preprocessing logic before executing its own required processing logic.

[0060] Taking the image processing unit set as an MFTs image processing chain as an example, a preprocessing unit independent of the MFTs can be added. This added preprocessing unit performs preprocessing on the image. Alternatively, the preprocessing logic can be set as an extension function of the first MFT in the MFTs image processing chain, with the first MFT responsible for executing the preprocessing logic. Of course, when designing the MFTs image processing chain, the first MFT in the MFTs chain can also be directly defined as an image pre-analysis processing MFT used only to execute preprocessing logic (without executing other logic outside of preprocessing). This first MFT provides some common / shared image pre-analysis processing functions and / or some non-shared but more basic image analysis processing functions, such as analyzing common information like the position, orientation, and brightness of faces in each frame, or analyzing basic but non-shared information like human contours, thus providing preprocessing functions for the image processing of other MFTs in the chain.

[0061] The preprocessing stage involves preprocessing the image to be processed, which may include, but is not limited to, processing related to face recognition, image processing, and / or processing related to system sensors. Table 1 below provides examples of further refinement processing types included in the above preprocessing aspects:

[0062] Table 1

[0063]

[0064] After obtaining the image to be processed, such as the image or video frame transmitted in real time by the camera in the image acquisition scene, preprocessing is first performed on it according to the set preprocessing logic, such as face analysis, human body contour analysis, brightness analysis, noise analysis, photosensitivity information analysis, etc., so that the common processing logic or some basic processing logic of some / all image processing units in the subsequent image processing chain can be completed uniformly in the preprocessing stage, avoiding multiple repeated executions of common processing logic by multiple image processing units in the subsequent image processing chain.

[0065] Step 203: Output the image to be processed and the preprocessing result information to the image processing unit set, so that the image processing unit set performs processing logic other than preprocessing in the corresponding processing logic of the image to be processed according to the preprocessing result information.

[0066] After preprocessing the image to be processed, the preprocessing results and the image to be processed are sent together to the image processing unit set for image processing.

[0067] When an image processing unit in the image processing unit set processes an image, if the processing logic of the image processing unit itself overlaps with the preprocessing logic of the preprocessing stage, the image processing unit does not need to execute the overlapping logic. Instead, it can directly extract the processing result information corresponding to the overlapping logic from the preprocessing result information and use it. Accordingly, it only executes the processing logic other than the overlapping part in the processing logic of the image processing unit. In other words, the image processing units in the image processing unit set execute the processing logic other than the preprocessing logic in the processing logic corresponding to the image processing unit itself only for the image to be processed, based on the preprocessing result information.

[0068] For common processing logic among different image processing units, executing the common processing logic once during the preprocessing stage can avoid multiple executions of the common processing logic by different image processing units.

[0069] For a chain-connected image processing unit set, that is, an image processing unit set that is an image processing chain, after the preprocessing of the image to be processed is completed, the image to be processed and the preprocessing result information can be sent together to the image processing unit at the head of the image processing chain, that is, to the first image processing unit of the image processing chain. Then, based on the connection order of each image processing unit in the chain, each image processing unit receives the information transmitted by its adjacent image processing unit in turn, and completes its own processing of the image to be processed based on the transmitted information, until the last processing unit in the chain, and the image output by the last processing unit is taken as the final result image of the image to be processed.

[0070] During the on-chain information transmission, the information transmitted by each image processing unit to its next adjacent image processing unit includes: the preprocessing result information, and the result image obtained after the image to be processed is transmitted to each image processing unit and processed by each image processing unit along the transmission path.

[0071] The above image processing process based on the image processing chain can be further implemented as follows:

[0072] 11) Use the image processing unit at the head of the image processing chain as the current image processing unit.

[0073] After completing the preprocessing of the image to be processed and sending the preprocessing result information and the image to be processed to the image processing unit at the head of the image processing chain, the image processing unit at the head of the image processing chain becomes the image processing unit that needs to process the image to be processed at the current progress, i.e., the current image processing unit.

[0074] 12) Using the current image processing unit, based on the corresponding information in the preprocessing result information, the receiving image is processed by the corresponding processing logic other than the preprocessing logic, and the processed image is used as the current intermediate image.

[0075] The information received by the current image processing unit includes two parts: one part is the preprocessing result information, and the other part is the result image after the original image to be processed is processed by each image processing unit on the transmission link when it is transmitted to the current image processing unit.

[0076] For the received information, the current image processing unit determines whether the preprocessing result information contains the information required for its image processing. If it does (that is, it indicates that the current image processing unit's own processing logic overlaps with the preprocessing logic), the required information is directly extracted from it, and based on the extracted information, the processing logic other than the preprocessing described in the processing logic corresponding to the current image processing unit is executed on the received image.

[0077] For example, the beautification unit extracts the coordinate sequence of the face and facial features from the preprocessing results, and performs a series of beautification processes on the face or facial features based on the extracted coordinate sequence, such as filter liquefaction, skin smoothing, color adjustment, and face slimming, without having to execute the face recognition processing logic before beautification processing; as another example, the background replacement / blurring unit extracts the human body contour coordinate sequence from the preprocessing results, and performs background replacement or background blurring processing based on the extracted coordinate sequence, without having to execute the human body contour recognition logic first.

[0078] It is easy to understand that if the preprocessing result information does not contain the information required for the image processing of the current image processing unit (that is, it indicates that the processing logic of the current image processing unit does not overlap with the preprocessing logic), then the current image processing unit executes the complete processing flow corresponding to its own processing logic.

[0079] 13) The current image processing unit outputs the preprocessing result information and the intermediate image it generates to the next image processing unit in the image processing chain.

[0080] In this embodiment, the image obtained by the current image processing unit after completing image processing of the received image is called the current intermediate image.

[0081] After completing the image processing of the received image, the current image processing unit transmits the processed image (i.e., the current intermediate image) along with the preprocessing result information to the next image processing unit in the chain.

[0082] 14) Update the current image processing unit based on the next image processing unit, and repeatedly execute the processing procedure for the current image processing unit until the image generated by the image processing unit at the end of the image processing chain is obtained, and use the image generated by the image processing unit at the end of the image processing chain as the processing result image of the image to be processed.

[0083] For example, in the image processing chain MFTs, through the above processing process, the preprocessing result information and the image obtained after the current MFT is processed are continuously passed to the next MFT. This allows each MFT in the MFTs chain to extract the required information from the passed preprocessing result information as needed and use it directly, thereby accelerating the processing speed of the image processing chain of the MFTs on the images output by the camera and reducing the total processing time.

[0084] Similarly, for image processing unit sets connected by other connection methods, such as those connected via a combination of parallel and chain / serial connections, the preprocessing result information and the processed result image obtained after passing the original image to be processed to the current unit node can be sequentially passed to the next unit (which may involve multiple parallel passes) according to the image processing timing reflected by the different connection relationships between the image processing units. This continues until the last image processing unit in the set is reached, at which point the image output by that last unit is used as the final result image of the image to be processed. Furthermore, for each image processing unit on the transmission path, if its execution logic overlaps with the preprocessing logic, the execution of the overlapping part is avoided; instead, the execution result information corresponding to the overlapping part is directly extracted from the transmitted preprocessing result information for use.

[0085] As can be seen from the above scheme, the image processing method disclosed in this application, after obtaining the image to be processed, performs preprocessing on the image to be processed to obtain preprocessing result information. The preprocessing performed includes at least: processing of the common processing logic representation of at least some image processing units in the image processing unit set; then, the image to be processed and the preprocessing result information are output to the image processing unit set, so that the image processing unit set performs processing logic other than the preprocessing in the corresponding processing logic of the image to be processed according to the preprocessing result information. This application extracts the common processing logic of at least some image processing units in the image processing unit set and adds a preprocessing step before the image processing of the image processing unit set. By placing the extracted common processing logic in the preprocessing step for execution, it can avoid the repetitive processing work caused by different units in the image processing unit set executing the common processing logic, thereby effectively reducing the total processing time of each image in the image processing unit set and avoiding or alleviating the problem of stuttering or frame dropping in the camera preview image.

[0086] Optionally, in one embodiment, when the preprocessing of the image to be processed is completed and the image to be processed and the preprocessing result information need to be sent to the image processing unit set, the image to be processed and the preprocessing result information are first fused to obtain a first fused image, and then the first fused image is output to the image processing unit set.

[0087] Similarly, in the process of image processing based on the image processing unit set, when the current image processing unit needs to output the preprocessing result information and its own generated intermediate image to the next image processing unit, it can also first perform fusion processing on the preprocessing result information and its own generated intermediate image to obtain a second fused image, and then output the second fused image to the next image processing unit of the current image processing unit. For example, the current MFT passes the second fused image obtained after fusion processing to the next MFT on the MFTs chain, etc.

[0088] By fusing images, both preprocessing results and the image data to be transmitted can be conveyed.

[0089] Optionally, the process of fusing the corresponding image and preprocessing result information can be further implemented as follows:

[0090] 21) Obtain a notification message in a preset format obtained by encapsulating the preprocessed result information.

[0091] This embodiment uses a preset format notification message to encapsulate the preprocessing result information. The message format includes multiple fields organized in a set order and size, including but not limited to a message header, message footer, message length, verification information, and message content.

[0092] An example message format for a notification message is shown in Table 2 below:

[0093] Table 2

[0094] message header Message length Message checksum Message content message tail 0xEF2D5B3C 32 bits 32 bits 8 bits * N 0xEF2D5B3D

[0095] The message content carries specific preprocessing result information. Table 3 below provides an example of the message content:

[0096] Table 3

[0097]

[0098] The message header and message footer serve as identifiers for a notification message, used to identify the message. The message length describes the length of the message content. The verification information is used to verify whether the notification message is a correct / valid message when decoding the message content from the fused image. Specifically, the message-related information can be calculated based on a preset verification algorithm (such as hash or message checksum), and the calculation result is compared with the verification information carried in the message. If they match, the message is considered correct / valid.

[0099] 22) Using preset bits of multiple different bytes in the byte information of the corresponding image, store the different bit information of the notification message in sequence to obtain the fused image of the corresponding image and the preprocessing result information.

[0100] The corresponding image is either the original image to be processed or a corresponding intermediate image.

[0101] After encapsulating the preprocessed result information into a message and obtaining the corresponding notification message, the notification message is fused with the corresponding image to be transmitted / transmitted (the original image to be processed or the corresponding intermediate image) based on a corresponding fusion algorithm (or overlay algorithm). Optionally, during the fusion process, preset bits of multiple different bytes in the byte information of the corresponding image to be transmitted are used to sequentially store different bit information of the notification message, thereby achieving the fusion of the two and obtaining the corresponding fused image. When storing the corresponding bit information of the notification message based on the preset bits, the original image information on those preset bits is overwritten or cleared.

[0102] Taking the corresponding image to be transmitted / transmitted as image Y as an example, assuming image Y is a 24-bit full-color RGB image of 1280*720, each pixel of Y has 3 bytes of RGB, and storing image Y requires a total of 2,764,800 bytes. Also assuming the total length of the notification message X is 1024 bytes (8192 bits), the following fusion / overlay algorithm can be used to fuse X and Y to obtain the corresponding fused image:

[0103] Each bit of message X is stored sequentially using the preset bits of each byte in multiple bytes of image Y. Assuming storage starts from the first byte of image Y, the preset bits of the first 8192 bytes of image Y are all covered by the content of message X, thus obtaining the fused image Z.

[0104] Preferably, the preset bit is the bit of the lower plane of the byte to which it belongs, for example, it can be the lowest bit of the byte to which it belongs (i.e., the bit of the zeroth plane) in order to minimize the loss of image information caused by the image participating in information fusion.

[0105] The applicant's analysis and verification process is further presented below:

[0106] The most commonly used color image is 24-bit full color. Each pixel in the image occupies 3 bytes of space, representing the RGB channels respectively. A total of 16,777,216 (256 to the power of 3) colors can be displayed, and the pixel value range is [0…255]. This can be represented by eight bits. Each bit represents a bit plane. There are eight bit planes in total, from the high bit to the low bit: the seventh bit plane, the sixth bit plane, the fifth bit plane…the zeroth bit plane.

[0107] For example, if the red pixel value at coordinate [0,0] in an image is 233, then its binary representation is 11101001 (0xE9), and its values ​​in each bit plane are as follows:

[0108] The red value of the seventh plane at coordinates [0,0] is 1;

[0109] The sixth plane has a red value of 1 at coordinates [0,0].

[0110] The fifth plane has a red value of 1 at coordinates [0,0].

[0111] The fourth plane has a red value of 0 at coordinates [0,0].

[0112] The third plane has a red value of 1 at coordinates [0,0].

[0113] The second plane has a red value of 0 at coordinates [0,0].

[0114] The first plane has a red value of 0 at coordinates [0,0].

[0115] The zeroth plane has a red value of 1 at coordinates [0,0].

[0116] The sum of the above eight bit plane values ​​is the red pixel value of the image at coordinates [0,0], which is 233. The specific summation formula is as follows:

[0117] 1*2^7+1*2^6+1*2^5+0*2^4+1*2^3+0*2^2+0*2^1+1*2^0=233.

[0118] Therefore, it can be seen that the weight of the seventh bit plane, which corresponds to the highest bit, is the largest, specifically 128, while the weight of the zero bit plane, which corresponds to the lowest bit, is the smallest, specifically 1.

[0119] Based on this, in this embodiment, the least significant bit of the corresponding byte (i.e., the bit of the zeroth plane) is preferably used as the preset bit of the byte to carry the information of the notification message. By selecting the least significant bit to carry the information of the notification message, the loss of detail information of the image participating in the fusion process is minimized. Taking red as an example, after occupying the least significant bit of its corresponding byte, its red pixel value changes from 233 to 232. Only the value of the zeroth plane is different. This negligible difference is indistinguishable to the human eye. The display effect of red pixel values ​​233 and 232 is compared as follows: Figure 3 As shown (color information is represented by corresponding grayscale), the human eye perceives no difference.

[0120] Accordingly, after receiving the fused image or the preprocessed output fused image from the previous image processing unit in the image processing unit set, the current image processing unit must first decode the fused image according to the corresponding decoding algorithm to obtain the preprocessing result information and the corresponding image to be processed (the original image to be processed or the intermediate image).

[0121] The decoding process based on the decoding algorithm includes:

[0122] 31) Extract the preset bit information of each byte of the received fused image, and concatenate the extracted information into the corresponding string in order.

[0123] Taking the least significant bit of the byte as a preset bit as an example, the least significant bit information of each byte of the fused image can be extracted sequentially starting from the first byte of the fused image, and the extracted least significant bit information can be concatenated into the corresponding string in the extraction order.

[0124] 32) Extract notification messages from the string according to the set extraction rules.

[0125] Optionally, the extraction rules can be specifically set as follows:

[0126] Determine whether the string contains a valid notification message. If it does, extract a segment of the string from the concatenated string, starting from the message header and ending at the message tail, based on the message header and message tail.

[0127] Specifically, the process can be based on header and trailer features to verify whether the concatenated string contains a valid header and trailer. If a valid header and trailer are found, further verification is performed based on checksum information to determine if the message is a valid notification message. For example, a message checksum is calculated based on the message length, and the calculated checksum is compared with the checksum provided in the message checksum information field. If they match, the string contains a valid notification message, and the notification message can be further parsed, extracting the preprocessing results from its message content field. Otherwise, if they do not match, the string does not contain a valid notification message. Specifically, the message length field and checksum field can be identified from the string between the header and trailer, and the required message length and checksum information (e.g., message checksum) can be read from them for message verification.

[0128] After extracting the notification message from the concatenated string, the notification message can be further removed from the fused image according to its corresponding position in the string, thereby obtaining the image to be processed by the current image processing unit (essentially the image obtained after removing the corresponding preset bit information of the original image to be processed or the intermediate image). Subsequently, based on the preprocessing result information carried in the notification message, other logics in its own processing logic other than the preprocessing logic can be executed on the image.

[0129] Taking image processing chain MFTs as an example, in conjunction with [see also...] Figure 4 Suppose the first MFT in the MFTs chain (e.g., Figure 4 In this context, MFTA (Image Pre-Analysis Processing) is defined as Image Pre-Analysis Processing (MFT), which provides some common or basic pre-analysis processing functions, such as analyzing common information like the position, orientation, and brightness of faces in each frame of the image. In addition, it also provides functions such as notification message generation and image overlay, as shown in Table 4.

[0130] Table 4

[0131]

[0132] The entire workflow of the MFTs chain is as follows:

[0133] For each frame in the image stream, the Pre-Analysis Processing (MFT), or MFTA, first performs preprocessing on the image, including face recognition, human contour information analysis, and brightness analysis. The preprocessed information is then formatted into a notification message X. This notification message X is overlaid onto the original image Y of that frame using an image overlay algorithm, generating a new image Z. MFTA outputs Z to MFTB. MFTB then decodes the notification message X from Z and uses the corresponding information in X to perform its own image processing on this frame (removing the overlapping parts between MFTB and MFTA). After image processing, MFTB uses the image overlay algorithm to overlay the notification message X onto the resulting image, generating a new image W, which is then passed to MFTC. Subsequent MFTs process images according to the steps of MFTB. If an MFT in the MFTs chain does not have overlapping processing logic with MFTA, then that MFT executes its own complete processing flow.

[0134] The following provides an example of an application scenario for the aforementioned MFTs chain.

[0135] Taking a camera image of 720p (1280*720) at 15fps as an example, the camera needs to send 15 frames of images per second, which means that the time for image processing of each frame is 1 / 15 of a second, approximately 66.7 milliseconds. In other words, all image processing tasks for each frame in the MFTs chain must be completed within 66.7 milliseconds.

[0136] Without adopting the scheme of this application, each MFT on the MFTs chain needs to perform face location analysis. The image processing work of 2 MFTs takes almost 55ms. Taking the maximum processing time of 66.7ms per frame as an example, the remaining image processing time of 11.7ms is almost insufficient to add other face image processing MFTs.

[0137] In this application, the pre-analysis processing MFT, i.e., MFTA, can be used to provide some common artificial intelligence (AI) and image pre-analysis data for other MFTs (B, C...N) in the image processing chain. Other MFTs in the image processing chain can directly use these analysis results data from the pre-processing stage to perform their own specific image processing work after excluding pre-processing logic, avoiding other MFTs outside of MFTA from performing repetitive image analysis work and saving the overall processing time for each frame of image.

[0138] Table 5 below further provides a comparison of image processing time in the above application scenarios with and without the proposed solution:

[0139] Table 5

[0140]

[0141] This embodiment of the application completes some common or basic analysis and processing of the image in the preprocessing stage, thereby avoiding repetitive image analysis tasks in each image processing unit and effectively shortening the total processing time of the image in the image processing unit set. Furthermore, this embodiment transmits the preprocessing result information along with the image according to the connection relationship of each unit in the image processing unit set, without the need to add communication processing mechanisms between image processing chains such as MFTs corresponding to the image processing unit set.

[0142] Alternatively, during the preprocessing stage, multiple preprocessing tasks without dependencies can be executed in parallel using multithreading to further reduce the total image processing time. It should be noted that for image processing unit sets with chained structures such as MFTs, the image processing process is carried out according to the transfer of images between different units on the chain. Different units need to process the images transferred to them according to the processing sequence under their chained structure. This makes it impossible for different units on the chain to perform parallel execution of processing tasks based on multithreading. This application improves the parallelism of image processing tasks by executing the extracted preprocessing logic in the preprocessing stage and executing different preprocessing tasks without dependencies in parallel using multithreading during the preprocessing stage. This further improves the processing efficiency of images on image processing chains such as MFTs and reduces their total processing time.

[0143] Corresponding to the image processing method described above, this application also discloses an image processing apparatus, the structure of which is as follows: Figure 5 As shown, it includes:

[0144] Preprocessing unit 501 is used to acquire an image to be processed, perform preprocessing on the image to be processed, obtain preprocessing result information, and output the image to be processed and the preprocessing result information to the image processing unit set;

[0145] Image processing unit set 502 is used to receive the image to be processed and the preprocessing result information output by the preprocessing unit, and to execute processing logic other than the preprocessing in the corresponding processing logic on the image to be processed according to the preprocessing result information.

[0146] The preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set.

[0147] In one embodiment, the image processing units included in the image processing unit set 502 are connected in a chain-like manner to form an image processing chain;

[0148] When the preprocessing unit 501 outputs the image to be processed and the preprocessing result information to the image processing unit set, it is specifically used to: output the image to be processed and the preprocessing result information to the image processing unit at the head of the image processing chain.

[0149] In one embodiment, when the preprocessing unit 501 outputs the image to be processed and the preprocessing result information to the image processing unit set, it is specifically used for:

[0150] The image to be processed and the preprocessing result information are fused to obtain a first fused image, which is then output to the image processing unit set.

[0151] In one embodiment, when the image processing unit set 502 executes processing logic other than preprocessing in the corresponding processing logic for the image to be processed based on the preprocessing result information, it is specifically used for:

[0152] The image processing unit at the head of the image processing chain is taken as the current image processing unit;

[0153] The current image processing unit uses the corresponding information in the preprocessing result information to perform processing logic other than preprocessing on the received image in the corresponding processing logic, and uses the processed image as the current intermediate image;

[0154] The current image processing unit outputs the preprocessing result information and the intermediate image it generates to the next image processing unit in the image processing chain.

[0155] The current image processing unit is updated based on the next image processing unit, and the processing procedure for the current image processing unit is executed cyclically until the image generated by the image processing unit at the end of the image processing chain is obtained. Then, the image generated by the image processing unit at the end of the image processing chain is used as the processing result image of the image to be processed.

[0156] In one embodiment, before outputting the preprocessing result information and the intermediate image generated by itself to the next image processing unit in the image processing chain, the current image processing unit in the image processing unit set 502 is further configured to: perform fusion processing on the preprocessing result information and the intermediate image generated by itself to obtain a second fused image, so as to output the second fused image to the next image processing unit in the image processing chain.

[0157] In one embodiment, when the preprocessing unit 501 or the image processing unit set 502 fuses the corresponding image and the preprocessing result information, it is specifically used for:

[0158] Obtain a notification message in a preset format obtained by encapsulating the preprocessed result information;

[0159] Using preset bits of multiple different bytes in the byte information of the corresponding image, the different bit information of the notification message is stored sequentially to obtain a fused image of the corresponding image and the preprocessing result information;

[0160] The corresponding image is either the image to be processed or the corresponding intermediate image.

[0161] In one implementation, the preset bit is the least significant bit of the byte.

[0162] In one embodiment, before the current image processing unit in the image processing unit set 502 performs processing logic other than preprocessing on the received image according to the corresponding information in the preprocessing result information, it is further configured to:

[0163] Extract the preset bit information of each byte of the received fused image, and concatenate the extracted information into the corresponding string in order;

[0164] The notification message is extracted from the string according to the set extraction rules.

[0165] The image processing apparatus disclosed in this application corresponds to the image processing methods disclosed in the above method embodiments, so the description is relatively simple. For related similarities, please refer to the descriptions of the above method embodiments, which will not be detailed here.

[0166] This application also discloses an electronic device, which may be, but is not limited to, a device in a variety of general or special computing device environments or configurations, such as: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0167] The composition and structure of electronic devices, such as Figure 6 As shown, it includes at least:

[0168] Memory 10 is used to store the computer instruction set;

[0169] Computer instruction sets can be implemented in the form of computer programs.

[0170] The processor 20 is configured to implement the image processing method disclosed in any of the above method embodiments by executing a set of computer instructions.

[0171] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices.

[0172] Electronic devices have a display device and / or have a display interface and can connect to an external display device.

[0173] Optionally, the electronic device may also include a camera assembly, and / or be connected to an external camera assembly.

[0174] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0175] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0176] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0177] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0178] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0179] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, the method comprising: Obtain the image to be processed; Preprocessing is performed on the image to be processed to obtain preprocessing result information; The preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set; the image processing units in the image processing unit set are connected to form an image processing chain. The image to be processed and the preprocessing result information are output to the image processing unit set, so that the image processing unit set performs processing logic other than preprocessing in the corresponding processing logic of the image to be processed according to the preprocessing result information; wherein, based on the connection order of each image processing unit in the image processing chain, each image processing unit receives the information transmitted by its adjacent image processing unit in sequence, and completes its own required processing of the image to be processed based on the transmitted information, until the last image processing unit in the image processing chain is reached, and the image output by the last image processing unit is taken as the processing result image of the image to be processed.

2. The method according to claim 1, wherein the image processing units included in the image processing unit set are connected in a chain-like manner to form an image processing chain.

3. The method according to claim 1, wherein outputting the image to be processed and the preprocessing result information to the image processing unit set comprises: The image to be processed and the preprocessing result information are fused to obtain a first fused image, which is then output to the image processing unit set.

4. The method according to claim 3, wherein fusing the image to be processed and the preprocessing result information to obtain a first fused image comprises: Obtain a notification message in a preset format obtained by encapsulating the preprocessed result information; By using preset bits of multiple different bytes in the byte information of the image to be processed, the different bit information of the notification message is stored sequentially to obtain the first fused image.

5. The method according to claim 1, wherein the image processing unit set performs processing logic other than the preprocessing in the corresponding processing logic on the image to be processed based on the preprocessing result information, including: The image processing unit at the head of the image processing chain is taken as the current image processing unit; The current image processing unit uses the corresponding information in the preprocessing result information to perform processing logic other than preprocessing on the received image in the corresponding processing logic, and uses the processed image as the current intermediate image; The current image processing unit outputs the preprocessing result information and the intermediate image it generates to the next image processing unit in the image processing chain. The current image processing unit is updated based on the next image processing unit, and the processing procedure for the current image processing unit is executed cyclically until the image generated by the image processing unit at the end of the image processing chain is obtained. Then, the image generated by the image processing unit at the end of the image processing chain is used as the processing result image of the image to be processed.

6. The method according to claim 5, further comprising, before using the current image processing unit to output the preprocessing result information and the intermediate image generated by itself to the next image processing unit in the image processing chain: The current image processing unit performs a fusion process on the preprocessing result information and the intermediate image it generates to obtain a second fused image, which is then output to the next image processing unit in the image processing chain.

7. The method according to claim 6, wherein fusing the preprocessed result information and the self-generated intermediate image to obtain a second fused image comprises: Obtain a notification message in a preset format obtained by encapsulating the preprocessed result information; By using preset bits of multiple different bytes in the byte information of the intermediate image, different bit information of the notification message is stored sequentially to obtain the second fused image.

8. The method according to claim 7, wherein the preset bit is the least significant bit of the byte to which it belongs.

9. The method according to claim 7, further comprising, before the step of using the current image processing unit to perform processing logic other than the preprocessing in the corresponding processing logic of the received image based on the corresponding information in the preprocessing result information, the method further comprising: The current image processing unit extracts preset bit information of each byte of the received fused image, and concatenates the extracted information into a corresponding string in sequence; The notification message is extracted from the string according to the set extraction rules.

10. An image processing apparatus, the apparatus comprising: The preprocessing unit is used to acquire the image to be processed, perform preprocessing on the image to be processed, obtain preprocessing result information, and output the image to be processed and the preprocessing result information to the image processing unit set. An image processing unit set is configured to receive the image to be processed and the preprocessing result information output by the preprocessing unit, and to execute processing logic other than preprocessing in the corresponding processing logic on the image to be processed according to the preprocessing result information; wherein, based on the connection order of each image processing unit in the image processing chain, each image processing unit sequentially receives information transmitted from its adjacent image processing unit above it, and completes its own required processing of the image to be processed based on the transmitted information, until the last image processing unit in the image processing chain is reached, at which point the image output by the last image processing unit is taken as the processing result image of the image to be processed; the image processing units included in the image processing unit set are connected to form an image processing chain; The preprocessing includes at least the processing of the common processing logic representation of at least some of the image processing units in the image processing unit set.

11. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor for implementing the image processing method as described in any one of claims 1-9 by calling and executing the instruction set stored in the memory.

Images