An image processing method, device and storage medium

Abstract

Embodiments of the present application provide an image processing method, device and storage medium. The first rotation angle required for accessing an AI processing engine and the rotation parameter value required for the AI processing result generated after the rotation of the AI processing engine can be automatically calculated for the to-be-processed image. In this way, the AI processing engine only needs to support a single image direction, without the need for compatibility with multiple image directions. The technical solution provided by the embodiments can automatically rotate the to-be-processed image to the image direction specified by the AI processing engine to support the normal processing work of the AI processing engine, which effectively reduces the cost of accessing the AI algorithm. Moreover, the AI processing result can be rotated to the rendering result position of the to-be-processed image based on the rotation parameter calculated for the AI processing result, which can ensure that the AI processing result is correctly rendered to the to-be-processed image, and further ensure that the data such as beautification executed based on the AI processing result can be correctly overlaid on the rendering result of the to-be-processed image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] Currently, in image processing scenarios such as beautification, it is often necessary to input images into AI algorithms so that AI algorithms can perform AI processing such as image segmentation and gesture recognition on the input images.

[0003] However, the sources of images used in AI algorithms are diverse, and the orientation of people in these images varies greatly. This requires AI algorithms to be compatible with various orientations. This requirement significantly increases the overall processing time of AI algorithms, and the development work required to enable different AI algorithms to support arbitrary orientations necessitates additional customization. All of these factors increase the cost of integrating AI algorithms and hinder the development efficiency of various products that require AI algorithm integration. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and storage medium in several aspects to reduce the cost of accessing AI algorithms.

[0005] This application provides an image processing method, including:

[0006] Calculate the first rotation angle required when the image to be processed is fed into the AI ​​processing engine;

[0007] The image to be processed is rotated according to the first rotation angle to obtain an access image, wherein the orientation of the target object in the access image is within the directional range specified by the AI ​​processing engine;

[0008] The access image is input into the AI ​​processing engine to obtain the AI ​​processing result corresponding to the access image;

[0009] Based on the rendering matrix associated with the image to be processed, calculate the rotation parameter values ​​required for the AI ​​processing result;

[0010] The AI ​​processing result is rotated according to the rotation parameter value so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

[0011] This application also provides a computing device, including a memory, a processor, and a communication component;

[0012] The memory is used to store one or more computer instructions;

[0013] The processor is coupled to the memory and the communication components, and is used to execute the one or more computer instructions for:

[0014] Calculate the first rotation angle required when the image to be processed is fed into the AI ​​processing engine;

[0015] The image to be processed is rotated according to the first rotation angle to obtain an access image, wherein the orientation of the target object in the access image is within the directional range specified by the AI ​​processing engine;

[0016] The access image is input into the AI ​​processing engine to obtain the AI ​​processing result corresponding to the access image;

[0017] Based on the rendering matrix associated with the image to be processed, calculate the rotation parameter values ​​required for the AI ​​processing result;

[0018] The AI ​​processing result is rotated according to the rotation parameter value so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

[0019] This application also provides a computer-readable storage medium for storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the aforementioned image processing method.

[0020] In this embodiment, the first rotation angle required for accessing the AI ​​processing engine and the rotation parameter values ​​required for the AI ​​processing result generated after the image is accessed by the AI ​​processing engine can be automatically calculated for the image to be processed. In this way, the AI ​​processing engine only needs to support a single image orientation, without needing to be compatible with multiple image orientations. The technical solution provided in this embodiment can automatically rotate the image to be processed to the image orientation specified by the AI ​​processing engine to support the normal processing operation of the AI ​​processing engine, which effectively reduces the cost of accessing the AI ​​algorithm. Furthermore, based on the rotation parameters calculated for the AI ​​processing result, the AI ​​processing result can be rotated to match the rendering result position of the image to be processed. This ensures that the AI ​​processing result is correctly aligned with the rendering result of the image to be processed, thereby ensuring that subsequent beautification and other data performed based on the AI ​​processing result can be correctly superimposed on the rendering result of the image to be processed. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1a Several exemplary images that require integration with AI algorithms are shown;

[0023] Figure 1b A schematic flowchart of an image processing method provided for an exemplary embodiment of this application;

[0024] Figure 1c An exemplary range of directions specified by an AI processing engine is shown;

[0025] Figure 2 A logical schematic diagram of an image processing method provided for an exemplary embodiment of this application;

[0026] Figure 3 A schematic diagram illustrating an application scenario of an image processing scheme provided as an exemplary embodiment of this application;

[0027] Figure 4 This is a schematic diagram of the structure of a computing device provided for another exemplary embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Figure 1a Several exemplary images requiring AI algorithm integration are shown. It can be seen that the orientation of the figures in these exemplary images varies. Currently, the requirement for AI algorithms to be compatible with various figure orientations leads to high integration costs, hindering the development efficiency of various products requiring AI algorithm integration. To address this, some embodiments of this application automatically calculate the first rotation angle required for integrating the AI ​​processing engine into the image to be processed, as well as the rotation parameter values ​​required for the AI ​​processing result generated after the image is integrated. In this way, the AI ​​processing engine only needs to support a single image orientation, without needing to be compatible with multiple image orientations. The technical solution provided in this embodiment can automatically rotate the image to be processed to the image orientation specified by the AI ​​processing engine to support the normal processing operation of the AI ​​processing engine, effectively reducing the cost of integrating AI algorithms. Furthermore, based on the rotation parameters calculated for the AI ​​processing result, the AI ​​processing result can be rotated to match the rendering result position of the image to be processed. This ensures that the AI ​​processing result is correctly aligned with the rendering result of the image to be processed, thereby ensuring that subsequent beautification and other data performed based on the AI ​​processing result can be correctly superimposed on the rendering result of the image to be processed.

[0030] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0031] Figure 1b This is a schematic flowchart of an image processing method provided as an exemplary embodiment of this application. Figure 2 This is a logical schematic diagram of an image processing method provided for an exemplary embodiment of this application. The method can be executed by an image processing apparatus, which can be implemented as a combination of software and / or hardware, and can be integrated into a computing device. (Reference) Figure 1b The method includes:

[0032] Step 100: Calculate the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine;

[0033] Step 101: Rotate the image to be processed according to the first rotation angle to obtain the access image. The direction of the target object in the access image is within the direction range specified by the AI ​​processing engine.

[0034] Step 102: Input the accessed image into the AI ​​processing engine to obtain the AI ​​processing result corresponding to the accessed image;

[0035] Step 103: Calculate the rotation parameter values ​​required for the AI ​​processing result based on the rendering matrix associated with the image to be processed;

[0036] Step 104: Rotate the AI ​​processing result according to the rotation parameter value so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

[0037] The image processing method provided in this embodiment can be applied to various scenarios that require access to AI algorithms, including but not limited to beautification scenarios, video editing scenarios, short video scenarios, live streaming scenarios, etc. This embodiment does not limit the application scenarios.

[0038] refer to Figure 1b and Figure 2In step 100, the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine can be calculated. The AI ​​processing engine may include several AI operators, such as face detection operators, image segmentation operators, gesture recognition operators, etc., which are not limited in this embodiment. In this embodiment, the AI ​​processing engine only needs to specify a single image direction, without needing to be compatible with multiple image directions. This greatly reduces the complexity of the AI ​​processing engine, allowing it to focus more on improving its accuracy and performance without being limited by image direction compatibility issues. The image direction can refer to the orientation of the directional marker line of a specific object in the image, such as the perpendicular bisector of the object. For example, it could be the orientation of the perpendicular bisector of a person in the image. In practical applications, the AI ​​processing engine can use the upward direction as the specified image direction. However, this embodiment is not limited to this; the AI ​​processing engine can also specify other image directions. For example, the bottom edge of the screen can be defined as the horizontal direction, and the angle between the upward direction and the horizontal direction is 90°. Of course, this is only an exemplary definition method, and this embodiment is not limited to this. The key here is that the AI ​​processing engine can specify only one image orientation without needing to be compatible with multiple image orientations. The specific orientation can be flexibly set according to actual needs. Based on the image orientation specified by the AI ​​processing engine, the required directional range of the target object in the image accessed by the AI ​​processing engine can be obtained. This directional range is typically a certain angle range taken on both sides of the image orientation specified by the AI ​​processing engine. Figure 1c This illustrates an exemplary range of directions specified by an AI processing engine. (Reference) Figure 1c For example, if the AI ​​processing engine specifies the image direction as the upward direction (e.g., defined as 90°), then an angle range of 60° can be taken to each side of the upward direction to obtain the direction range specified by the AI ​​processing engine [30°, 150°]. Thus, the direction of the target object in the image input to the AI ​​processing engine needs to be within this direction range; otherwise, the AI ​​processing engine will not be able to perform AI processing correctly. In this embodiment, the target object can be a person, an animal, or even a building or household item, etc., and this embodiment is not limited to these. In practical applications, the orientation of the top of the target object is usually used as the direction of the target object; for example, the orientation of the top of a person's head can be used as the direction of the person. Of course, other methods can be specified to define the direction of the target object, and this embodiment is not limited to this.

[0039] In one optional implementation: N candidate rotation angles can be pre-set. Based on these, rotation operations can be performed on the image to be processed according to each of the N candidate rotation angles to obtain the first rotated image corresponding to each of the N candidate rotation angles. The first rotation angle is determined according to the orientation of the target object in each of the first rotated images and the directional range specified by the AI ​​processing engine. Specifically, during the traversal of the N candidate rotation angles, if the orientation of the target object in the first rotated image is within the directional range specified by the AI ​​processing engine, then the currently traversed candidate rotation angle can be determined as the first rotation angle. N can be an integer not less than 4. A preferred value for N is 4, under which the four candidate rotation angles can be set to 0°, 90°, 180°, and 270° respectively. Thus, the image to be processed can be rotated four times according to these four candidate rotation angles to find the candidate rotation angle that adapts the rotated image to the image orientation specified by the AI ​​processing engine, which is then used as the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine.

[0040] In this implementation, an exemplary search scheme may be as follows: For the first rotated image corresponding to each of the N candidate rotation angles, define the position regions that fit the directional range specified by the AI ​​processing engine; calculate the face confidence score corresponding to each image key point in each position region; determine the sum of the face confidence scores corresponding to each image key point in each position region as the confidence score parameter corresponding to the candidate rotation angle; and determine the candidate rotation angle with the largest corresponding confidence score parameter among the N candidate rotation angles as the first rotation angle. That is, under each candidate rotation angle, it can be determined whether the image key points corresponding to the target object in the rotated image to be processed are located within the directional range specified by the AI ​​processing engine. If so, that is, the confidence score is large enough, it can be confirmed that a suitable first rotation angle has been found. To this end, in this exemplary search scheme, object detection algorithms, such as various face detection algorithms, can be introduced to calculate the object confidence score corresponding to the image key points required in the search scheme. The object confidence score of the image key points is used to characterize the confidence score of the image key points belonging to the specified object. It is a common parameter in various object detection algorithms, and the calculation logic of this parameter will not be described in detail here. For example, when the target object is a person, the upper half of the rotated image to be processed can be defined at each candidate rotation angle as the location area that matches the directional range specified by the AI ​​processing engine. Based on this, the face confidence score corresponding to each key point in the image within this location area can be calculated and the confidence scores can be summed. By comparing the sums of confidence scores calculated at each candidate rotation angle, the candidate rotation angle with the highest sum of confidence scores can be found and used as the first rotation angle corresponding to the image to be processed.

[0041] In addition, in this embodiment, the following optimization schemes can be proposed for the calculation process of the first rotation angle:

[0042] Predetermined direction priority scheme: In this optimization scheme, if the image to be processed is a video frame in a video, then if the difference between the sum of the first and second candidate rotation angles ranked first in the confidence parameter is less than a preset range, it is determined whether the first rotation angle used in the previous video frame exists among these two candidate rotation angles; if it does, then the first rotation angle used in the previous video frame is used as the first rotation angle corresponding to the image to be processed. That is, in the process of calculating the first rotation angle for subsequent video frames, if the first rotation angle used in the previous video frame has not been eliminated (for example, it will be eliminated if the confidence sum is too low), the first rotation angle used in the previous video frame will be used first.

[0043] Interval Calculation Scheme: In this optimized scheme, if the image to be processed is a video frame, the calculation of the first rotation angle required for the image to be processed to be accessed by the AI ​​processing engine can be performed once every specified number of frames, instead of performing the calculation for every single frame in the video. This effectively reduces the computational load in step 100. Furthermore, in this optimized scheme, for video frames within the interval corresponding to the specified number of frames, the most recently calculated first rotation angle can be reused.

[0044] Immediate Error Correction Solution: In this optimization solution, if an incompatibility issue is found with the first rotation angle used within the aforementioned interval, its use is stopped, and the operation of calculating the first rotation angle required for the image to be processed when inputting into the AI ​​processing engine is performed on the video frames within the interval. Incompatibility refers to the use of the first rotation angle causing incorrect processing of the rotated image after input into the AI ​​processing engine. Incompatibility issues are usually discovered in subsequent beautification and other processing stages, and the discovered incompatibility issues can be fed back to step 100 in this embodiment. In this way, incompatibility issues can be detected promptly during the use of the first rotation angle, and the operation of calculating the first rotation angle can be immediately performed on the relevant video frames to correct the incompatibility issue in a timely manner. Furthermore, it should be understood that in this optimization solution, the limitations of the aforementioned predetermined direction optimization solution and interval calculation solution can be removed, and the operation of calculating the first rotation angle can be performed immediately. For example, if the success rate of detecting faces in m consecutive frames is found to be greater than 0% and less than 50%, the operation of calculating the first rotation angle can be performed immediately on the subsequent video frames instead of using the original first rotation angle.

[0045] Asynchronous execution scheme: In this optimized scheme, a dedicated thread is started for step 100 outside the main process, so that step 100 is executed asynchronously with other steps. This avoids step 100 affecting the frame rate of video processing. In the beautification scenario, the main process can be the beautification main process, which, in addition to handling steps 101, 102, and 104 in this embodiment, can also handle related beautification processing tasks.

[0046] First-frame waiting solution: In this optimized solution, the main process waits for a dedicated thread to provide the first rotation angle required for the image to be processed to be input into the AI ​​processing engine before performing the rotation operation on the image to be processed according to the first rotation angle and the operation of inputting the input image into the AI ​​processing engine. Especially for the first frame of a video, it cannot inherit any prior first rotation angle and can only calculate the first rotation angle in real time. Therefore, the main process can wait for the dedicated thread to calculate the first rotation angle corresponding to the first frame of the video before starting the relevant steps to ensure that the first frame of the video can be processed normally by the AI ​​processing engine. In beautification scenarios, this effectively avoids problems such as the loss of beautification effects in the first frame of image mode and video editing, and beautification effect distortion.

[0047] Continue to refer to Figure 1b and Figure 2 In step 101, the image to be processed can be rotated according to the first rotation angle to obtain the access image. It should be understood that the orientation of the target object in the access image will be within the directional range specified by the AI ​​processing engine; that is, the orientation of the target object in the access image will match the image orientation specified by the AI ​​processing engine. In this way, the AI ​​processing engine can run the relevant AI operators according to its specified image orientation without needing to concern itself with the image orientation of the access image.

[0048] In step 102, the accessed image can be input into the AI ​​processing engine to obtain the corresponding AI processing result. The AI ​​processing result can be a beautification result, a special effects result, etc. Step 102 involves the processing stage of the AI ​​processing engine, which is not the focus of this embodiment. The AI ​​processing engine can process the accessed image according to various current or future AI operators, which will not be detailed here. The AI ​​processing result includes one or more of the following: facial key points, human body segmentation regions, gesture recognition results, or human body key points.

[0049] Continue to refer to Figure 1b and Figure 2 In step 103, the rotation parameter values ​​required for the AI ​​processing result can be calculated based on the rendering matrix associated with the image to be processed.

[0050] Here, the rendering matrix is ​​usually specified by the image provider or can be obtained from other specified channels. Each image to be processed has a corresponding rendering matrix. The rendering matrix associated with the image to be processed is usually set according to the rotation, scaling, translation, and other operations to be performed on the image to be processed. That is, using the rendering matrix, the rotation, scaling, translation, and other operations on the image to be processed can be implemented to render the image to be processed into the required state, thereby obtaining the rendering result. For example, when a video frame captured by a mobile phone camera is used as the image to be processed, a rendering matrix can be preset for the video frame. After rendering the video frame according to the rendering matrix, the video frame can be rendered as a style that is reduced by 2 times, rotated by 30°, and shifted to the left by 30ms to obtain the rendering result of the video frame. In this case, all video frames captured by the mobile phone follow the same rendering matrix. Of course, a rendering matrix can also be specifically specified for certain video frames. This embodiment does not limit this. Obviously, the rendering result of the image to be processed will usually be different from its initial state. Of course, in many cases, the image to be processed may be rendered directly according to the initial state. In these cases, the rendering result of the image to be processed will be consistent with its initial state.

[0051] In step 103, the required rotation parameter values ​​can be calculated for the AI ​​processing results corresponding to the accessed image. Step 103 can also be executed by the aforementioned dedicated thread, and the calculation results can be fed back to the main process to avoid affecting the processing efficiency of the main process.

[0052] As explained above regarding the rendering matrix, the image to be processed undergoes rotation, scaling, and translation operations to produce the rendering result. Although the input image is also obtained by rotating the image to be processed, this rotation operation usually does not achieve the exact rotation effect caused by the rendering matrix. Consequently, the AI ​​processing result generated from the input image also does not achieve the rotation effect caused by the rendering matrix. This leads to a mismatch between the position of the AI ​​processing result and the rendering result of the image to be processed.

[0053] Therefore, in step 103, the required rotation parameter values ​​can be calculated for the AI ​​processing result corresponding to the accessed image. These rotation parameter values ​​may include, but are not limited to, a second rotation angle and a flip axis. Thus, the rotation parameter values ​​can be used to characterize the angle at which the AI ​​processing result needs to be rotated, whether a flip is required, and the flip axis if a flip is required.

[0054] In one optional implementation: the texture coordinates corresponding to the image to be processed can be calculated based on the rendering matrix associated with the image to be processed; based on the mapping relationship between the texture coordinates and candidate rotation angles and rotation parameter values, the rotation parameter value that matches the texture coordinates and first rotation angle of the image to be processed can be found. The texture coordinates are typically equal to the product of the rendering matrix and the original texture coordinates; that is, the texture coordinates can be obtained by rotating, scaling, and / or translating the original texture coordinates according to the rendering matrix. It is understood that the AI ​​processing result in this embodiment can be used as a texture, and the texture coordinates corresponding to the image to be processed are calculated in order to map the texture onto the rendered image to be processed. The texture coordinates can then be used to indicate which position in the AI ​​processing result each vertex in the rendered image to be processed should be sampled from. The original texture coordinates, like the rendering matrix, can be specified and directly obtained; therefore, the texture coordinates of the image to be processed can be calculated without difficulty, and will not be detailed here. In this field, texture coordinates are used to define the mapping relationship between textures and graphics. Each vertex in the graphics is associated with a texture coordinate, indicating that the vertex needs to read texture image data from that coordinate position. The original texture coordinates are the original texture coordinates set for the image to be processed.

[0055] In addition, in this implementation, a mapping relationship between texture coordinates, candidate rotation angles, and rotation parameter values ​​can be pre-built. In this way, when the texture coordinates and first rotation angle of the image to be processed are determined, the rotation parameter value corresponding to this combination can be found from the mapping relationship.

[0056] Considering the diverse sources of the images to be processed, and the potential differences in origin positions between images from different sources, and given that the rendering process references both the rendering matrix and the origin position, different origin positions can significantly impact the final rendering result. Therefore, this implementation introduces the origin position as another lookup criterion. This allows for the pre-establishment of a mapping relationship between the origin position, texture coordinates, candidate rotation angles, and rotation parameter values. Based on this, this implementation: determines the origin position of the image to be processed; and, based on the mapping relationship between the origin position, texture coordinates, candidate rotation angles, and rotation parameter values, finds rotation parameter values ​​that match the origin position, texture coordinates, and rotation angle of the image to be processed.

[0057] During their research, the inventors discovered that the values ​​for the origin position, texture coordinates, and candidate rotation angles are all finite. Therefore, this embodiment proposes to exhaustively enumerate the combinations of origin position, texture coordinates, and candidate rotation angles. Under each combination, with the goal of rotating the AI ​​processing result of the image to match the rendering result corresponding to the rendering matrix of the image to be processed, the rotation parameter values ​​occurring during the rotation process are collected. A correlation is established between each combination and its corresponding rotation parameter values ​​to generate a mapping relationship. Here, "matching" can be interpreted as alignment. For example, the eyeliner texture included in the AI ​​processing result should be aligned and cover the root of the human eyelashes in the rendered image to be processed.

[0058] The origin position typically has only two possibilities: it is located in the lower left corner or in the upper left corner. To facilitate the construction of mapping relationships, in practical applications, two identifiers can be introduced for the origin position: a true identifier and a false identifier. The true identifier is used to indicate that the origin position is located in the lower left corner, and the false identifier is used to indicate that the origin position is located in the upper left corner.

[0059] Texture coordinates can include eight cases: uv = (x, y), uv = (x, 1.0 - y), uv = (1.0 - x, y), uv = (1.0 - x, 1.0 - y), uv = (y, x), uv = (y, 1.0 - x), uv = (1.0 - y, x), and uv = (1.0 - y, 1.0 - x). Texture coordinates typically have two axes, U and V, hence the name UV coordinates, where U represents the distribution on the horizontal axis and V represents the distribution on the vertical axis.

[0060] As mentioned earlier, the candidate rotation angle can include N cases, which corresponds to the N candidate rotation angles mentioned earlier.

[0061] In this way, 2*8*N combinations can be formed, and when N is 4, 64 combinations can be constructed.

[0062] In this embodiment, during the pre-setting of the mapping relationship, AI processing results are generated for each combination, and the rotation parameter values ​​generated during the rotation are collected with the goal of rotating the AI ​​processing results of the image to match the rendering results corresponding to the current rendering matrix. In this way, corresponding rotation parameter values ​​can be collected for each combination, thereby establishing a mapping relationship between the combination and the rotation parameter values. It should be understood that during the pre-setting of the mapping relationship, the image used for testing is also processed. Figure 1bSteps 100-103 and the corresponding rendering operations of the rendering matrix were performed; various rotation attempts were made on the AI ​​processing results to ultimately adapt the AI ​​processing results to the position between the AI ​​processing results and the rendering results. Here, position adaptation refers to the overlap of the positions of relevant points between the two, for example, the facial key points in the AI ​​processing results overlap one-to-one with the facial key points in the rendering results.

[0063] In this way, based on these preset mapping relationships, the rotation parameter values ​​required for the AI ​​processing results can be quickly and accurately found in step 103.

[0064] Building upon this, in step 104, the AI ​​processing result can be rotated according to the rotation parameter values ​​to ensure that the position of the rotated AI processing result matches the position of the rendered result generated by rendering the image to be processed according to the rendering matrix. As mentioned earlier, positional adaptation here refers to making the relevant points in the rotated AI processing result overlap with the relevant points in the rendered result by rotating the AI ​​processing result. Furthermore, the rotation operation on the AI ​​processing result can include, but is not limited to, rotation and flipping along a flip axis.

[0065] Furthermore, after completing step 104, in this embodiment, a beautification operation can also be performed based on the rotated AI processing result. This allows the relevant points in the AI ​​processing result to be accurately superimposed onto the rendering result of the image to be processed, ensuring the accuracy of the display position of the beautification result and improving the beautification effect.

[0066] In summary, this embodiment can automatically calculate the first rotation angle required for the image to be processed to access the AI ​​processing engine, as well as the rotation parameter values ​​required for the AI ​​processing result generated after the image is accessed by the AI ​​processing engine. In this way, the AI ​​processing engine only needs to support a single image orientation, without needing to be compatible with multiple image orientations. The technical solution provided in this embodiment can automatically rotate the image to be processed to the image orientation specified by the AI ​​processing engine to support the normal processing operation of the AI ​​processing engine, which effectively reduces the cost of accessing the AI ​​algorithm. Furthermore, based on the rotation parameter values ​​calculated for the AI ​​processing result, the AI ​​processing result can be rotated to match the rendering result position of the image to be processed. This ensures that the AI ​​processing result is correctly aligned with the rendering result of the image to be processed, thereby ensuring that subsequent beautification and other data performed based on the AI ​​processing result can be correctly superimposed on the rendering result of the image to be processed.

[0067] Figure 3 This is a schematic diagram illustrating an application scenario of an image processing scheme, provided as an exemplary embodiment of this application. (Reference) Figure 3The target object in the image to be processed is a person. Assuming the image direction specified by the AI ​​processing engine is the upward direction, the direction range required by the AI ​​processing engine can be set to the angle range obtained by taking 60° on each side of the upward direction.

[0068] Based on this, Figure 3 The image processing scheme is represented by two cases: In the case of the dashed box on the left, the image to be processed is a single picture. In this case, the origin is usually located in the upper left, and the corresponding label of the origin is false. In the case of the dashed box on the right, the image to be processed can be a video frame in a video. The source of the video can be a real-time shot by a camera or a video that has been edited. In this case, the origin is usually located in the lower left, and the corresponding label of the origin is true.

[0069] Figure 3 The document also illustrates an exemplary rendering matrix. After rendering the image to be processed according to the rendering matrix and two origin positions, two rendering results can be obtained (see reference). Figure 3 (The rendering result within the two dashed boxes). Additionally... Figure 3 The example also shows the texture coordinates corresponding to the rendering matrix, for example, uv = (y, 1.0 - x).

[0070] According to the technical solution provided in this embodiment, four candidate rotation angles can be provided in advance: 0°, 90°, 180°, and 270°. Referring to the rotation angle calculation scheme provided in the previous embodiment, the rotation angle corresponding to the image to be processed can be calculated to be 90°. In this way, after rotating the image to be processed by 90°, the direction of the person in it can conform to the direction range specified by the AI ​​processing engine.

[0071] After the image is input into the AI ​​processing engine, it can generate AI processing results such as facial key points, human body key points, and gesture recognition results. The direction of the AI ​​processing results is completely consistent with the direction of the input image.

[0072] Then, based on the preset mapping relationship between the origin position, the first rotation angle, the texture coordinate combination, and the rotation parameter values, it can be used to... Figure 3 The combination of [false, 90°, (y, 1.0-x)] within the dashed box on the left yields the mapped rotation parameter value, which is [180°, no flip axis]. Figure 3 The combination of [true, 90°, (y, 1.0-x)] in the case of the dashed box on the right can be used to find the rotation parameter value of the mapping, which is [180°, flip axis Y-axis].

[0073] Thus, a 180° rotation of the AI ​​processing result within the left dashed box will align the AI ​​processing result with the rendered result 1. However, within the right dashed box, a Y-axis flip followed by a 180° rotation is required to align the AI ​​processing result with the rendered result 1.

[0074] As can be seen, based on the image processing scheme provided in this embodiment, the first rotation angle required for accessing the AI ​​processing engine can be automatically calculated for the image to be processed, and the required rotation parameter value can be automatically calculated for the AI ​​processing result output by the AI ​​processing engine. This ensures that the image orientation of the image caused by the input AI processing is correct, and the generated AI processing result can be rotated to match the position of the rendering result. This ensures the accuracy and efficiency of image processing, and the AI ​​processing engine no longer needs to be compatible with various image orientations, thereby reducing the access cost of the AI ​​processing engine.

[0075] It should be noted that some processes described in the above embodiments and accompanying drawings include multiple operations appearing in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should also be noted that the descriptions such as "first" and "second" in this document are used to distinguish different rotation angles, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0076] Figure 4 This is a schematic diagram of the structure of a computing device provided for another exemplary embodiment of this application. For example... Figure 4 As shown, the computing device includes: memory 40 and processor 41.

[0077] Processor 41, coupled to memory 40, is used to execute computer programs in memory 40 for:

[0078] Calculate the first rotation angle required when the image to be processed is fed into the AI ​​processing engine;

[0079] The image to be processed is rotated according to the first rotation angle to obtain the access image. The orientation of the target object in the access image is within the directional range specified by the AI ​​processing engine.

[0080] The input image is fed into the AI ​​processing engine to obtain the corresponding AI processing result for the input image;

[0081] Based on the rendering matrix associated with the image to be processed, calculate the rotation parameter values ​​required for the AI ​​processing result;

[0082] The AI ​​processing result is rotated according to the rotation parameter values ​​so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

[0083] In an optional embodiment, the processor 41, during the process of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine, can be used to:

[0084] Obtain N candidate rotation angles;

[0085] Perform rotation operations on the image to be processed according to N candidate rotation angles to obtain the first rotated image corresponding to each of the N candidate rotation angles;

[0086] The first rotation angle is determined based on the orientation of the target object in each of the first rotated images and the directional range specified by the AI ​​processing engine.

[0087] In an optional embodiment, the target object is a person, and the processor 41, in determining the first rotation angle based on the orientation of the target object in each first rotated image and the directional range specified by the AI ​​processing engine, can be used to:

[0088] For each of the N candidate rotation angles, the corresponding first rotation image is delineated to fit the position region specified by the AI ​​processing engine.

[0089] Calculate the face confidence score for each image key point in each location region;

[0090] The sum of the face confidence scores corresponding to each image key point within the location area is used as the confidence parameter corresponding to the candidate rotation angle.

[0091] Among the N candidate rotation angles, the candidate rotation angle with the highest corresponding confidence parameter is determined as the first rotation angle.

[0092] In an optional embodiment, the processor 41 is further configured to:

[0093] If the image to be processed is a video frame in a video, then if the difference between the sum of the first and second candidate rotation angles ranked first in the confidence parameter is less than a preset range, it is determined whether the first and second candidate rotation angles contain the first rotation angle used in the previous video frame.

[0094] If it exists, then the first rotation angle used in the previous video frame is taken as the first rotation angle corresponding to the image to be processed.

[0095] In an optional embodiment, the processor 41 is further configured to:

[0096] If the image to be processed is a video frame in a video, the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine is performed once every specified number of frames;

[0097] For video frames within the interval corresponding to the specified frame number, the most recently calculated first rotation angle is used.

[0098] If the first rotation angle used within the interval is not suitable, then stop using it and perform the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine for the video frames in the interval.

[0099] In one alternative embodiment, a dedicated thread is started outside the main process; the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine is performed by the dedicated thread.

[0100] The operation of rotating the image to be processed according to the first rotation angle, and the operation of inputting the input image into the AI ​​processing engine, are executed by the main process after obtaining the first rotation angle provided by the dedicated thread.

[0101] In one optional embodiment, N is 4, and the four candidate rotation angles are 0°, 90°, 180°, and 270°.

[0102] In an optional embodiment, the processor 41, during the process of calculating the rotation parameter values ​​required for the AI ​​processing result based on the rendering matrix associated with the image to be processed, can be used to:

[0103] Calculate the texture coordinates of the image to be processed based on the rendering matrix associated with the image to be processed;

[0104] Based on the mapping relationship between texture coordinates and candidate rotation angles and rotation parameter values, find the rotation parameter values ​​that match the texture coordinates and first rotation angle of the image to be processed.

[0105] In an optional embodiment, during the process of finding a rotation parameter value that matches the texture coordinates and the first rotation angle corresponding to the image to be processed, based on the mapping relationship between texture coordinates and candidate rotation angles and rotation parameter values, the processor 41 can:

[0106] Determine the origin position of the image to be processed;

[0107] Based on the mapping relationship between the origin position, texture coordinates, and candidate rotation angles and rotation parameter values, find rotation parameter values ​​that match the origin position, texture coordinates, and first rotation angle of the image to be processed.

[0108] In an optional embodiment, the processor 41 may, during the process of constructing the mapping relationship, be used to:

[0109] The combination of origin position, texture coordinates and candidate rotation angles is exhaustively explored. Under each combination, the rotation parameter values ​​that occur during the rotation process are collected with the goal of rotating the AI ​​processing result of the image to match the rendering result corresponding to the rendering matrix.

[0110] Establish a relationship between each combination and its corresponding rotation parameter value to generate a mapping relationship.

[0111] In an alternative embodiment, the rotation parameter values ​​include a second rotation angle and a flip axis.

[0112] In one optional embodiment, the AI ​​processing results include one or more of the following: facial key points, human body segmentation regions, gesture recognition results, or human body key points.

[0113] Furthermore, such as Figure 4 As shown, the computing device also includes other components such as a communication component 42 and a power supply component 43. Figure 4 The components shown are only schematically illustrated and do not imply that the computing device includes only the components shown in Figure 4.

[0114] It is worth noting that the technical details of the above embodiments of the computing device can be referred to the relevant descriptions in the foregoing method embodiments. To save space, they will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0115] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can perform the steps that can be executed by a computing device in the above method embodiments.

[0116] The above Figure 4 The memory 40 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0117] The above Figure 4The communication component 42 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0118] The above Figure 4 The power supply component 43 provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, comprising: Calculate the first rotation angle required when the image to be processed is fed into the AI ​​processing engine; The image to be processed is rotated according to the first rotation angle to obtain an access image, wherein the orientation of the target object in the access image is within the directional range specified by the AI ​​processing engine; The access image is input into the AI ​​processing engine to obtain the AI ​​processing result corresponding to the access image; Calculate the texture coordinates corresponding to the image to be processed based on the rendering matrix associated with the image to be processed; Determine the origin position of the image to be processed; The combination of origin position, texture coordinates and N candidate rotation angles is exhaustively enumerated. Under each combination, with the goal of rotating the AI ​​processing result of the image to match the rendering result corresponding to the rendering matrix, the rotation parameter values ​​that occur during the rotation process are collected. Establish a correlation between each combination and its corresponding rotation parameter values ​​to generate a mapping relationship; Based on the mapping relationship, find rotation parameter values ​​that match the origin position, texture coordinates and first rotation angle of the image to be processed; The AI ​​processing result is rotated according to the rotation parameter value so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

2. The method according to claim 1, wherein calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine includes: Obtain N candidate rotation angles; The image to be processed is rotated according to the N candidate rotation angles to obtain the first rotated image corresponding to each of the N candidate rotation angles; The first rotation angle is determined based on the orientation of the target object in each of the first rotated images and the directional range specified by the AI ​​processing engine.

3. The method according to claim 2, wherein the target object is a person, and the step of determining the first rotation angle based on the orientation of the target object in each of the first rotated images and the directional range specified by the AI ​​processing engine includes: For each of the N candidate rotation angles corresponding to the first rotation image, a position region adapted to the directional range specified by the AI ​​processing engine is defined. Calculate the face confidence score corresponding to each image key point in each of the aforementioned location regions; The sum of the face confidence scores corresponding to each image key point within the location area is determined as the confidence parameter corresponding to the candidate rotation angle. The candidate rotation angle with the highest confidence parameter among the N candidate rotation angles is determined as the first rotation angle.

4. The method according to claim 3, further comprising: If the image to be processed is a video frame in a video, then if the difference between the sum of the first and second candidate rotation angles ranked first in the confidence parameter is less than a preset range, it is determined whether the first and second candidate rotation angles contain the first rotation angle used in the previous video frame. If it exists, then the first rotation angle used in the previous video frame is taken as the first rotation angle corresponding to the image to be processed.

5. The method according to claim 1, further comprising: If the image to be processed is a video frame in a video, then the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine is performed once every specified number of frames; For video frames within the interval corresponding to the specified frame number, the most recently calculated first rotation angle is used; If the first rotation angle used within the interval is not suitable, then stop using it and perform the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine on the video frames in the interval.

6. The method according to claim 1, wherein a dedicated thread is started outside the main process; the operation of calculating the first rotation angle required when the image to be processed is accessed by the AI ​​processing engine is executed by the dedicated thread; The operation of rotating the image to be processed according to the first rotation angle, and the operation of inputting the input image into the AI ​​processing engine, are executed by the main process after obtaining the first rotation angle provided by the dedicated thread.

7. The method according to claim 2, wherein N is 4, and the four candidate rotation angles are 0°, 90°, 180°, and 270°.

8. The method according to claim 1, wherein the rotation parameter values ​​include a second rotation angle and a flip axis.

9. The method according to claim 1, wherein the AI ​​processing result includes one or more of the following: facial key points, human body segmentation regions, gesture recognition results, or human body key points.

10. A computing device, comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is used to execute the one or more computer instructions for: Calculate the first rotation angle required when the image to be processed is fed into the AI ​​processing engine; The image to be processed is rotated according to the first rotation angle to obtain an access image, wherein the orientation of the target object in the access image is within the directional range specified by the AI ​​processing engine; The access image is input into the AI ​​processing engine to obtain the AI ​​processing result corresponding to the access image; Calculate the texture coordinates corresponding to the image to be processed based on the rendering matrix associated with the image to be processed; Determine the origin position of the image to be processed; The combination of origin position, texture coordinates and N candidate rotation angles is exhaustively enumerated. Under each combination, with the goal of rotating the AI ​​processing result of the image to match the rendering result corresponding to the rendering matrix, the rotation parameter values ​​that occur during the rotation process are collected. Establish a correlation between each combination and its corresponding rotation parameter values ​​to generate a mapping relationship; Based on the mapping relationship, find rotation parameter values ​​that match the origin position, texture coordinates and first rotation angle of the image to be processed; The AI ​​processing result is rotated according to the rotation parameter value so that the position of the rotated AI processing result is adapted to the rendering result generated by rendering the image to be processed according to the rendering matrix.

11. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the image processing method according to any one of claims 1-9.

Images