Image processing method and device
By obtaining the feature information of the human head, adjusting the size and angle of the preset model, the problem that decorative components cannot be accurately matched in the prior art is solved, and intelligent image processing and precise decoration are realized.
Patent Information
- Application Number
- CN202510404528.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-27
- Publication Date
- 2025-07-22
AI Technical Summary
When the prior art virtual decoration of special parts of the human head, pre-designed decorations cannot accurately match the real position and angle, resulting in complex operation and low intelligence.
By obtaining feature information of the target part in the image, including the inclination angle and position, adjusting the size and inclination angle of the preset model to match it with the target part, accurately decorative parts are generated.
It improves the effect and intelligence of image processing, simplifies the operation process, and enables decorative components to accurately match specific parts of the human head.
Smart Images
Figure CN120355820A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and particularly to an image processing method and apparatus. Background Art
[0002] With the development of image processing technology, more and more image processing software and video processing software can further process users' photos or videos. For example, virtual dressing for the portraits in images is a relatively common image decoration method.
[0003] Currently, in terms of rendering a specific part of the human head, such as facial rendering, only a pre-designed decoration is added at a general position of the head contour, without considering information such as the real position, angle, and contour of this specific part in the image. Therefore, the pre-designed decoration cannot well decorate this specific part, and often the size and position of the pre-designed decoration need to be manually adjusted to match the real part. The existing processing method is relatively complex and has a low degree of intelligence. Summary of the Invention
[0004] An embodiment of this application provides an image processing method, which improves the effect and intelligence degree of image processing by adaptively adjusting a preset model.
[0005] In a first aspect, an embodiment of this application provides an image processing method, and the method includes:
[0006] Obtain a target part in an image;
[0007] Obtain feature information of the target part, where the feature information includes an inclination angle and a position;
[0008] Determine a preset model corresponding to the target part, where the shape structure of the preset model matches the shape structure of the target part;
[0009] Adjust the size and inclination angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decoration component matching the target part.
[0010] In a second aspect, an embodiment of this application provides a target part matching apparatus, and the apparatus includes an obtaining unit and a processing unit, where:
[0011] The obtaining unit is configured to obtain a target part in an image;
[0012] The obtaining unit is further configured to obtain feature information of the target part, where the feature information includes an inclination angle and a position;
[0013] The processing unit is configured to determine a preset model corresponding to the target part, and the shape structure of the preset model matches the shape structure of the target part;
[0014] The processing unit is further configured to adjust the size and tilt angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative component that matches the target part.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of the embodiments of the present application.
[0016] In a fourth aspect, an embodiment of the present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored on a memory through the data interface and executes the methods in the first aspect and any optional implementation manner described above.
[0017] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application.
[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.
[0019] It can be seen that in the embodiments of the present application, the electronic device obtains the target part in the image and acquires the feature information of the target part, where the feature information includes the tilt angle and the position. Further, the preset model is adjusted according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative component that matches the target part. By using the present application, the preset model can be adaptively adjusted based on the feature information of the target part in the image, and the adjusted preset model can be used to generate a decorative component that matches the target part, thereby improving the image processing effect and the degree of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.
[0021] Figure 1 It is a schematic flowchart of an image processing method provided by an embodiment of the present application;
[0022] Figure 2 It is a schematic flowchart of another image processing method provided by an embodiment of the present application;
[0023] Figure 3 It is a schematic structural diagram of key point annotation of a preset model provided by an embodiment of the present application;
[0024] Figure 4 It is a schematic structural diagram of the minimum bounding rectangle of a preset model provided by an embodiment of the present application;
[0025] Figure 5 It is a schematic diagram of the aspect ratio of a preset model at different rotation angles provided by an embodiment of the present application;
[0026] Figure 6 It is a schematic diagram of the scaling of a preset model provided by an embodiment of the present application;
[0027] Figure 7 It is a schematic diagram of the functional units of a target part matching device provided by an embodiment of the present application;
[0028] Figure 8 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners
[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0030] In the description, claims, and the above-mentioned drawings of this application, terms such as "first", "second", etc. are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.
[0031] Reference to "embodiments" herein means that specific features, structures, or characteristics described in connection with the embodiments can be included in the embodiments of this application. The phrase appears in various places in the description and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.
[0032] In current image processing solutions, when adding virtual decorations to a specific part of the human head, the pre-designed decorations cannot well modify this specific part. In many cases, it is also necessary to manually adjust the size and position of the pre-designed decorations to match the real ears, and the operation process is relatively complex and the degree of intelligence is low.
[0033] In view of the above problems, the embodiments of this application provide an image processing method, which is applied to an electronic device. The following is a detailed introduction with reference to the drawings.
[0034] The above-mentioned electronic devices can include, for example, distributed storage servers, traditional servers, large storage systems, desktop computers, laptop computers, tablet computers, handheld computers, smart phones, portable digital players, smart watches, and smart bracelets, etc.
[0035] OpenCV: An open-source algorithm library for image processing and computer vision. It can implement some common algorithms related to image processing. In the embodiments of this application, OpenCV is used to identify images or videos to implement the function of extracting the target part.
[0036] First, please refer to Figure 1 , Figure 1 which is a schematic flowchart of an image processing method provided by the embodiments of this application. This method may include but is not limited to the following steps:
[0037] 101. Obtain the target part in the image.
[0038] Specifically, when an electronic device recognizes an image in a video to obtain a target part, any one or more of the following methods can be adopted. The methods include: Open CV, R-CNN, Fast R-CNN, Faster R-CNN, ION, HyperNet, SDP-CRC, YOLO, G-CNN, SSD, etc. For example, obtaining feature parts in a face image, such as ears, eyes, lips, nose, hair, facial contour, etc., or they can also be feature parts of the human body, such as clothing, accessories, hands, feet, legs, etc.
[0039] 102. Obtain the feature information of the target part, where the feature information includes the tilt angle and the position.
[0040] Specifically, after the electronic device obtains the target part, it can further extract the feature information of the target part.
[0041] Optionally, obtaining the feature information of the target part, where the feature information includes the tilt angle and the position, includes: framing the target part in the form of a target box to obtain a target box; obtaining the vertex coordinates of the target box; obtaining the tilt angle and the position of the target part according to the vertex coordinates of the target box; and using the tilt angle and the position as the feature information of the target part.
[0042] Specifically, taking the extraction of the feature information of the target part by an electronic device using Open CV as an example, such as extracting ears, the ear detector of Open CV recognizes the approximate position coordinates of the ears in the image. The example pseudo-code process is as follows:
[0043] Cascade Classifier ear Detector("haarcascade_mcs_leftear.xml");
[0044] Ear Detector detect Multi Scale(gray, ears);
[0045] The result returned by the ear detector is the size and position information of the rectangular box containing the ears (target part). The position information of the rectangular box includes the coordinates [xCV, yCV] of the upper left corner point. Thus, according to the width widthCV of the rectangular box and the height heightCV of the rectangular box, the coordinates of the other three vertices of the rectangular box can be determined, and the center point of the user's ears can be calculated. The center point coordinates can be recorded as [xCV + widthCV / 2, yCV + heightCV / 2]. The size of the rectangular box can be determined by the width and height of the rectangular box.
[0046] In addition, the inclination angle of the target part, i.e., the inclination angle of the rectangular frame, can also be calculated based on the size and position information of the rectangular frame.
[0047] 103. Determine a preset model corresponding to the target part, where the shape and structure of the preset model match the shape and structure of the target part.
[0048] Specifically, there are multiple preset models. Different target parts correspond to different preset models. For example, there is a preset model of the nose for the nose, a preset model of the ear for the ear, a preset model of the eye for the eye, etc. And the shape and structure framework of the preset model match the shape and structure framework of the target part.
[0049] 104. Adjust the size and inclination angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative component that matches the target part.
[0050] Specifically, it can be understood that the preset model is a model pre-trained by the electronic device and can be used to further adjust the decorative component of the target part. For example, the preset model of the eye, the preset model of the ear, the preset model of the nose, etc. And the electronic device needs to make a more refined adjustment to the preset model to make it better match the target part.
[0051] Optionally, the adjusting the size and inclination angle of the preset model according to the feature information to obtain an adjusted preset model includes: rotating the preset model according to the inclination angle of the target part so that the inclination angle of the rotated preset model matches the inclination angle of the target part; determining the size of the target part according to the vertex position of the target part; scaling the size of the preset model according to the size of the target part so that the size of the scaled preset model matches the size of the target part.
[0052] Specifically, the basis for the electronic device to adjust the preset model is the feature information of the target part. For example, the width of the rectangle of the target part is 5 cm, the height is 10 cm, and the tilt angle is 45 degrees to the left. Then, the electronic device can perform a rotation process on the preset model based on this information to adjust the tilt angle of the preset model so that the tilt angle of the preset model after the rotation process matches the tilt angle of the target part, that is, to make it have the same tilt angle as the target part, or the deviation of the tilt angle meets a preset range, such as 5 degrees, 2 degrees, 1 degree, 3 degrees, etc. If the tilt angle is 45 degrees to the left, after the preset model is rotated counterclockwise by 40 degrees, 50 degrees, 42 degrees, etc., it can be understood that the tilt angle of the preset model after the rotation process matches the tilt angle of the target part. The tilt angle can specifically include the tilt angle value and the tilt direction. Determine the size of the target part according to the vertex position of the target part, that is, determine the size of the target part according to the positions of the four vertices of the target box containing the target part in the coordinate system. That is, determine the size of the target part according to the coordinates of the four vertices of the target box. And scale the size of the preset model according to the size of the target part so that the size of the preset model after the scaling process matches the size of the target part, that is, the size of the adjusted preset model can be scaled down to the lowest degree in proportion on the basis of completely matching the target part.
[0053] In addition, the tilt angle can include the tilt degree and the tilt direction. Taking the above tilt angle of 45 degrees to the left as an example, the left tilt is the tilt direction, and 45 degrees is the tilt degree. The tilt angle and the tilt direction can also be separated. That is, the left tilt is the tilt direction, and 45 degrees is the tilt angle. In short, the tilt angle can include the tilt direction or not.
[0054] It can be seen that in the embodiment of the present application, the electronic device obtains the target part in the image and acquires the feature information of the target part, and the feature information includes the tilt angle and the position. Further, the preset model is adjusted according to the feature information to obtain the adjusted preset model, where the adjusted preset model is used to generate a decoration part matching the target part. By using the present application, the preset model can be adaptively adjusted based on the feature information of the target part in the image, and the adjusted preset model can be used to generate a decoration part matching the target part, thereby improving the image processing effect and the degree of intelligence.
[0055] Consistent with the above Figure 1 shown embodiment, please refer to Figure 2 , Figure 2 is a schematic flowchart of another image processing method provided by the embodiment of the present application, which is applied to an electronic device. The method includes:
[0056] 201. Acquire the target part in the image.
[0057] 202. Obtain the feature information of the target part, where the feature information includes the tilt angle and the position.
[0058] 203. Determine the preset model corresponding to the target part, where the shape structure of the preset model matches the shape structure of the target part.
[0059] 204. Adjust the size and tilt angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative component that matches the target part.
[0060] Steps 201 - 204 refer to steps 101 - 104 and will not be elaborated here.
[0061] 205. Generate a decorative component that matches the target part according to the adjusted preset model.
[0062] Specifically, it can be understood that for the adjusted preset model of the electronic device, its tilt angle and size basically meet the contour of the target part. After successfully matching the target part, the electronic device can further generate a decorative component that matches the target part on this basis. For example, decorative components with different colors and shapes, realizing the possibility of virtual dressing of the target part in multiple solutions and in all directions.
[0063] In addition, some image texture and key point recognition algorithms also require prior information such as the position, angle, and size of the target object. The method in the embodiments of the present application can also be used for reference for processing.
[0064] 206. Translate the decorative component to the position of the target part in the image so that the decorative component matches the target part.
[0065] Specifically, referring to the position of the target part recognized by OpenCV, translate the above-mentioned decorative component to the corresponding position of the target position, so that the decorative component generated based on the preset model just covers the target part in the image or video. Its translation formula can be as follows:
[0066] [XnNew, YnNew] = [Xn, Yn] + [XCV + widthCV / 2, YCV + heightCV / 2];
[0067] Among them, XnNew is the abscissa after translation of any point n of the decoration component, and YnNew is the ordinate after translation of point n. Xn is the abscissa before translation of point n, and Yn is the ordinate before translation of point n. XCV + widthCV / 2 is the translation size of the abscissa, and YCV + heightCV / 2 is the translation size of the ordinate. Among them, XCV is the abscissa of the upper left vertex of the target box of the target part, and YCV is the ordinate of the upper left vertex of the target box of the target part.
[0068] It can be seen that in the embodiment of the present application, the electronic device obtains the target part in the image and acquires the feature information of the target part, where the feature information includes the tilt angle and the position. Further, the preset model is adjusted according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decoration component that matches the target part. The decoration component is translated to the position of the target part in the image so that the decoration component matches the target part. By using the present application, the preset model is adaptively adjusted based on the feature information of the target part in the image. The adjusted preset model can be used to generate a decoration component that matches the target part and make the decoration component match the target part. Thereby improving the effect and intelligence level of image processing.
[0069] In a possible example, before adjusting the size and tilt angle of the preset model according to the feature information, the method further includes: performing key point annotation on the preset model to obtain a plurality of key points; determining the minimum bounding rectangle of the preset model and acquiring the initial coordinates of the four vertices of the minimum bounding rectangle; calculating the coordinates of the center point of the preset model according to the initial coordinates of the four vertices; establishing a coordinate system with the center point as the coordinate origin; calculating the target coordinates of each of the key points of the preset model in the coordinate system; the adjusting the size and tilt angle of the preset model according to the feature information includes: adjusting the target coordinates of each of the key points of the preset model in the coordinate system according to the feature information to adjust the size and tilt angle of the preset model.
[0070] Specifically, taking the human ear model as an example of the preset model, as Figure 3 shown, the electronic device first needs to preset a human ear model to be used. The electronic device performs key point annotation on the preset model. It can be annotated with 68 key points, or 100, or 50. The number of key points depends on the complexity of the contour of the preset model, etc. 68 key points form a contour in the shape of an ear. And with these 68 key points as the skeleton, any desired pattern or shape can be added to make virtual dressing when generating the decoration component later. Then as Figure 4As shown, determine the minimum bounding rectangle of the preset model. And obtain the initial coordinates of the four vertices of the minimum bounding rectangle; set the coordinates of the nth marked point as [xn, yn]. Calculate the minimum values of all marked point coordinates as xmin and ymin, and the maximum values as xmax and ymax. Then the coordinates of the four vertices of the rectangle are [xmin, ymin], [xmax, ymin], [xmin, ymax], and [xmax, ymax] respectively. The calculation method of the width width-model of the rectangle is width-model = xmax - xmin, and the calculation method of the height height-model is height-model = ymax - ymin. The center point of the preset model is the geometric center point of the rectangle, as Figure 4 shown in the annotation, and its coordinates are [(xmin + xmax) / 2, (ymin + ymax) / 2].
[0071] Furthermore, since the current coordinate system is established with the coordinates of the upper left vertex of the rectangle as the coordinate origin, the horizontal axis of this coordinate system is the x-axis and increases sequentially from left to right; the vertical axis is the y-axis and increases sequentially from top to bottom. To facilitate further adjustment of the preset model later, after obtaining the coordinates of the center point, a coordinate system is established with the coordinates of the center point as the coordinate origin, and based on this, the target coordinates of multiple other key points are obtained. That is, the entire preset model is translated to the position where the center point coincides with the coordinate origin.
[0072] Furthermore, adjusting the size and tilt angle of the preset model according to the feature information can be understood as adjusting the target coordinates of each key point of the preset model in this coordinate system. By adjusting the target coordinates, the adjustment of the size and tilt angle of the preset model is realized.
[0073] It can be seen that the electronic device performs multiple key point annotations on the preset model, which is convenient for determining the minimum bounding rectangle of the preset model and the initial coordinates of the four vertices of the minimum bounding rectangle. Further, the center point coordinates of the rectangle are used as the center point coordinates of the preset model. Further, a coordinate system is established with the center point as the coordinate origin, and the target coordinates of the multiple key points are re-obtained according to the coordinate system, and finally the position of the preset model is reset to zero. And based on the adjustment of the target coordinates of multiple key points, the adjustment of the size and tilt angle of the preset model is realized.
[0074] In a possible example, calculating the target coordinates of each key point of the preset model in the coordinate system includes: obtaining the initial coordinates of each key point among the multiple key points; calculating the target coordinates of each key point according to the initial coordinates of each key point and the coordinates of the center point.
[0075] Specifically, after the electronic device establishes a coordinate system with the center point as the coordinate origin, it re-obtains the target coordinates of the multiple key points according to the coordinate system, that is, calculates the coordinates of each key point after translation. After obtaining the center point of the preset model, using this center point as a reference point, the entire model framework is translated to the coordinate origin. The translated coordinate system still has the horizontal axis as the x-axis, increasing sequentially from left to right; the vertical axis as the y-axis, increasing sequentially from top to bottom. The translation method is: subtracting the center point [(xmin + xmax) / 2, (ymin + ymax) / 2] from the coordinates [xn, yn] of any one of the multiple key points of the preset model. That is: [xnNew, ynNew] = [xn, yn] - [(xmin + xmax) / 2, (ymin + ymax) / 2].
[0076] It can be seen that when the electronic device translates the preset model, it calculates the target coordinates of each key point according to the initial coordinates of each key point and the coordinates of the center point, so as to determine the translation unit and the translation direction, and thus complete the translation operation of the preset model. This simplifies the operation steps and improves the accuracy and speed of translation.
[0077] In a possible example, the obtaining the inclination angle and position of the target part according to the vertex coordinates of the target box includes: determining the position of the target part according to the vertex coordinates of the target box, and calculating the aspect ratio of the target box according to the vertex coordinates of the target box; calculating the aspect ratios of the preset model at different inclination angles respectively; if the aspect ratio of the preset model matches the aspect ratio of the target box when the inclination angle of the preset model is the first angle value, then taking the first angle value as the inclination angle of the target part.
[0078] Specifically, after the electronic device obtains the vertex coordinates of the target box, since the target box completely encloses the target part, it can be understood that the target box is the minimum circumscribed rectangle of the target part. Therefore, determining the coordinates of each vertex of the target box determines the position of the target part. Obviously, it is very easy for the electronic device to calculate the aspect ratio of the target box according to the vertex coordinates of the target box. For example, width = the absolute value of the difference in the abscissas of the vertices symmetric about the ordinate, height = the absolute value of the difference in the ordinates of the vertices symmetric about the abscissa, and then the aspect ratio can be further calculated. The aspect ratio of the target box can be denoted as widthCV / heightCV.
[0079] In addition, when the electronic device calculates the tilt angle of the target part, taking the ear as an example of the target part. Since in the image, the tilt angle of the ear may be different from the tilt angle of a preset model such as an ear model. It can be assumed that both are in the same plane, and only the angular difference in the two-dimensional space is considered. Since the ear belongs to the side of the human face, the relative positions of the eyes and the nose cannot be used as a reference to correct the problem of inconsistent angles. According to the calculation of the rectangular frame of the preset model, by rotating the preset model by different angles, different rectangular frames can be obtained, that is, the aspect ratios of the rectangular frames are different. As Figure 5 shown, the left figure is the preset model at the initial tilt angle, and the right figure is the effect of rotating the preset model clockwise by 45 degrees. The aspect ratio of the circumscribed rectangular frame of the preset model at the initial tilt angle is 0.5196, while for the preset model rotated by 45 degrees, its rectangular aspect ratio is 0.6752. The different aspect ratios correspond to different rotation angles of the model. The process of coordinate transformation for any point rotation of the preset model is as follows:
[0080]
[0081] where θ is the rotation angle. (x, y) is the original coordinate of a certain key point of the preset model at the initial tilt angle, and (xnew, ynew) is the coordinate of this certain key point after rotation by the angle θ.
[0082] Therefore, after the electronic device knows that the aspect ratio of the rectangular frame of the minimum circumscribed rectangle of the target part is widthCV / heightCV, it sequentially rotates the preset model clockwise by 0 - 90 degrees and calculates the aspect ratios of the corresponding matrices. Find the situation closest to the value of widthCV / heightCV, and the reverse corresponding rotation angle is the rotation angle that the preset model needs to be adjusted, that is, the tilt angle of the target part. For example, when the preset model is rotated by 50 degrees, its aspect ratio is closest to the aspect ratio of the rectangular frame of the target frame, then 50 degrees is the first angle value and is the tilt angle of the target part. Rotate the coordinates of the preset model according to the first angle value, and the width and height of the corresponding new rectangular frame are: width-model-rot and height-model-rot. In the embodiments of the present application, the example of clockwise rotation is used for illustration. If the counterclockwise rotation angle needs to be determined, this method can also be referred to. Whether the preset model rotates counterclockwise or clockwise can be either that the electronic device detects the rotation direction input by the user or that the electronic device detects the deflection angle of the human head in the image, and is determined by this deflection angle.
[0083] It can be seen that the electronic device determines the position of the target part according to the vertex coordinates of the target box, and calculates the aspect ratio of the target box according to the vertex coordinates of the target box; calculates the aspect ratios of the preset model at different tilt angles respectively. The tilt angle when the aspect ratio of the preset model matches the aspect ratio of the target box is used as the tilt angle of the target part. The process of determining the tilt angle and the target part is simplified, and the calculation efficiency is improved.
[0084] In a possible example, the scaling the size of the preset model according to the size of the target part includes: obtaining the target box corresponding to the target part, and obtaining the width value of the target part and the height value of the target part; obtaining the width value of the preset model and the height value of the preset model; calculating the width ratio of the target part to the preset model and the height ratio of the target part to the preset model respectively; determining the larger value of the width ratio and the height ratio, and then scaling the size of the preset model according to the larger value.
[0085] Specifically, when the electronic device scales the size of the preset model according to the size of the target part, a scaling factor needs to be calculated. Still taking the target part as the ear as an example, in order to make the size of the preset model match the size of the ear in the image, the preset model needs to be scaled proportionally. The principle of proportional scaling is to make the square of the preset model completely match the target box containing the target part, and the area of the minimum circumscribed rectangle of the scaled preset model is the smallest. Referring to the rectangular box of the rotated preset model and the rectangular box detected by OpenCV, the scaling factor is calculated as follows:
[0086] scaleX = widthCV / width-model-rot;
[0087] scaleY = heightCV / height-model-rot;
[0088] scale = max{scaleX, scaleY};
[0089] Among them, scaleX is the scaling ratio of the abscissa, widthCV is the width of the target box, and width-model-rot is the width of the minimum bounding rectangle of the rotated preset model; scaleY is the scaling ratio of the ordinate, heightCV is the height of the target box, and height-model-rot is the height of the minimum bounding rectangle of the rotated preset model; scale is the scaling factor, and max{scaleX, scaleY} is to select the larger value of scaleX and scaleY as the scaling factor. That is, in order to ensure that the aspect ratio of the preset model after scaling remains unchanged and can fully match the target box, the larger value of the scaling ratio of the abscissa and the scaling ratio of the ordinate is selected as the scaling factor to scale the preset model after rotation adjustment. Of course, the order of first scaling the preset model or performing the scaling process is not limited. Here, the example is to first perform the rotation process and then perform the scaling process for illustration.
[0090] Furthermore, after obtaining the scaling factor, scale the coordinates of any point of the preset model and calculate the rectangle of the preset model after scaling. That is:
[0091]
[0092] Among them, x is the abscissa of a key point of the preset model before scaling, and y is the ordinate of this key point of the preset model before scaling; xnew is the abscissa of this key point of the preset model after scaling, and ynew is the ordinate of this key point of the preset model after scaling.
[0093] The entire scaling process is as Figure 6 shown. Figure 6 The numbers and circles on the left represent multiple key points of the preset model, the dashed box represents the minimum bounding rectangle of the preset model, and the solid box represents the target box of the target part recognized by Open CV. The star point in the rectangle represents the center point of the rectangle. Here, the center points of the two rectangles coincide. That is, the center point of the minimum bounding rectangle of the preset model coincides with the center point of the target box. Comparing the sizes of the dashed box and the solid box, scale the coordinates of the preset model according to the scaling factor scale to obtain the preset model as shown on the Figure 6 right side.
[0094] It can be seen that when scaling the preset model, the electronic device first calculates the scaling factor, and on the premise of ensuring that the preset model after scaling can fully match the target box, performs the minimum degree of scaling, that is, selects the maximum value of the scaling ratios of the horizontal and vertical coordinates to scale the preset model, ensuring that the ratio of the preset model after scaling remains unchanged. While ensuring the scaling effect, the effect of the preset model matching the target part is improved accordingly.
[0095] Please refer toFigure 7 FIG. 1 is a schematic diagram of the functional units of a target part matching device 700 according to an embodiment of the present invention. The original video recognition device 700 according to an embodiment of the present invention can be an in-built device of the above-mentioned electronic device or an external device of the above-mentioned electronic device.
[0096] In an implementation of the device according to an embodiment of the present invention, the device 700 includes an acquisition unit 410 and a processing unit 420, where:
[0097] The acquisition unit 410 is configured to acquire a target part in an image;
[0098] The acquisition unit 410 is further configured to acquire feature information of the target part, where the feature information includes an inclination angle and a position;
[0099] The processing unit 420 is configured to determine a preset model corresponding to the target part, where the shape structure of the preset model matches the shape structure of the target part;
[0100] The processing unit 420 is further configured to adjust the size and inclination angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative part matching the target part.
[0101] In a possible example, before the processing unit 420 adjusts the size and inclination angle of the preset model according to the feature information, the processing unit 420 is further configured to perform key point annotation on the preset model to obtain a plurality of key points; determine the minimum circumscribed rectangle of the preset model, and acquire the initial coordinates of the four vertices of the minimum circumscribed rectangle; calculate the center point coordinates of the preset model according to the initial coordinates of the four vertices; establish a coordinate system with the center point as the coordinate origin; calculate the target coordinates of each of the key points of the preset model in the coordinate system; and adjusting the size and inclination angle of the preset model according to the feature information includes: adjusting the target coordinates of each of the key points of the preset model in the coordinate system according to the feature information to adjust the size and inclination angle of the preset model.
[0102] In a possible example, in terms of calculating the target coordinates of each of the key points of the preset model in the coordinate system, the processing unit 420 is specifically configured to acquire the initial coordinates of each of the key points in the plurality of key points; and calculate the target coordinates of each of the key points according to the initial coordinates of each of the key points and the coordinates of the center point.
[0103] In a possible example, when obtaining the feature information of the target part, where the feature information includes the tilt angle and the position, the obtaining unit 410 is specifically configured to frame the target part in the form of a target box to obtain a target box; obtain the vertex coordinates of the target box; obtain the tilt angle and the position of the target part according to the vertex coordinates of the target box; and use the tilt angle and the position as the feature information of the target part.
[0104] In a possible example, when obtaining the tilt angle and the position of the target part according to the vertex coordinates of the target box, the processing unit 420 is specifically configured to determine the position of the target part according to the vertex coordinates of the target box, and calculate the aspect ratio of the width to the height of the target box according to the vertex coordinates of the target box; calculate the aspect ratios of the width to the height of the preset model at different tilt angles respectively; if the aspect ratio of the preset model matches the aspect ratio of the target box when the tilt angle of the preset model is a first angle value, then use the first angle value as the tilt angle of the target part.
[0105] In a possible example, when adjusting the size and the tilt angle of the preset model according to the feature information to obtain an adjusted preset model, the processing unit 420 is specifically configured to rotate the preset model according to the tilt angle of the target part so that the tilt angle of the rotated preset model matches the tilt angle of the target part; determine the size of the target part according to the vertex position of the target part; and scale the size of the preset model according to the size of the target part so that the size of the scaled preset model matches the size of the target part.
[0106] In a possible example, when scaling the size of the preset model according to the size of the target part, the processing unit 420 is specifically configured to obtain the target box corresponding to the target part, and obtain the width value of the target part and the height value of the target part; obtain the width value of the preset model and the height value of the preset model; calculate the width ratio of the target part to the preset model and the height ratio of the target part to the preset model respectively; determine the larger value of the width ratio and the height ratio, and then scale the size of the preset model according to the larger value.
[0107] In a possible example, after the processing unit 420 adjusts the size and the tilt angle of the preset model according to the feature information to obtain an adjusted preset model, it is further configured to generate a decorative component matching the target part according to the adjusted preset model; and translate the decorative component to the position of the target part in the image so that the decorative component matches the target part.
[0108] Please refer to again Figure 8 which is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device according to the embodiment of the present invention corresponds to the electronic device described above. The electronic device includes a power supply module and other structures, and includes a processor 801, a storage device 802, and a communication interface 803. Data can be exchanged between the processor 801, the storage device 802, and the communication interface 803.
[0109] The storage device 802 may include a volatile memory, such as a random-access memory (RAM); the storage device 802 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 802 may further include a combination of the above types of memories.
[0110] The processor 801 may be a central processing unit (CPU). In one embodiment, the processor 801 may also be a Graphics Processing Unit (GPU). The processor 801 may also be a combination of a CPU and a GPU. In one embodiment, the storage device 802 is used to store program instructions. The processor 801 may call the program instructions to perform the following steps:
[0111] Obtain the target part in the image;
[0112] Obtain the feature information of the target part, where the feature information includes an inclination angle and a position;
[0113] Determine a preset model corresponding to the target part, where the shape structure of the preset model matches the shape structure of the target part;
[0114] Adjust the size and inclination angle of the preset model according to the feature information to obtain an adjusted preset model, where the adjusted preset model is used to generate a decorative part matching the target part.
[0115] In a possible example, before the processor 801 adjusts the size and tilt angle of the preset model according to the feature information, it is further configured to perform key point annotation on the preset model to obtain multiple key points; determine the minimum bounding rectangle of the preset model, and obtain the initial coordinates of the four vertices of the minimum bounding rectangle; calculate the coordinates of the center point of the preset model according to the initial coordinates of the four vertices; establish a coordinate system with the center point as the coordinate origin; re-obtain the target coordinates of the multiple key points according to the coordinate system; the adjusting the size and tilt angle of the preset model according to the feature information includes: adjusting the target coordinates of each key point of the preset model in the coordinate system according to the feature information, so as to adjust the size and tilt angle of the preset model.
[0116] In a possible example, in terms of calculating the target coordinates of each key point of the preset model in the coordinate system, the processor 801 is specifically configured to: obtain the initial coordinates of each key point in the multiple key points; calculate the target coordinates of each key point according to the initial coordinates of each key point and the coordinates of the center point.
[0117] In a possible example, in terms of obtaining the feature information of the target part, the feature information includes the tilt angle and the position, the processor 801 is specifically configured to: frame the target part in the form of a target box to obtain a target box; obtain the vertex coordinates of the target box; obtain the tilt angle and the position of the target part according to the vertex coordinates of the target box; use the tilt angle and the position as the feature information of the target part.
[0118] In a possible example, in terms of obtaining the tilt angle and the position of the target part according to the vertex coordinates of the target box, the processor 801 is specifically configured to: determine the position of the target part according to the vertex coordinates of the target box, and calculate the aspect ratio of the target box according to the vertex coordinates of the target box; calculate the aspect ratios of the preset model at different tilt angles respectively; if the aspect ratio of the preset model matches the aspect ratio of the target box when the tilt angle of the preset model is the first angle value, then use the first angle value as the tilt angle of the target part.
[0119] In a possible example, the processor 801 adjusts the size and tilt angle of the preset model according to the feature information to obtain an adjusted preset model, and specifically is configured to: rotate and process the preset model according to the tilt angle of the target part so that the tilt angle of the rotated and processed preset model matches the tilt angle of the target part; determine the size of the target part according to the vertex position of the target part; scale and process the size of the preset model according to the size of the target part so that the size of the scaled and processed preset model matches the size of the target part.
[0120] In a possible example, when scaling and processing the size of the preset model according to the size of the target part, the processor 801 is specifically configured to: obtain a target box corresponding to the target part, and obtain the width value and height value of the target part; obtain the width value and height value of the preset model; calculate the width ratio between the target part and the preset model and the height ratio between the target part and the preset model respectively; determine the larger value between the width ratio and the height ratio, and then scale and process the size of the preset model according to the larger value.
[0121] In a possible example, after the processor 801 adjusts the size and tilt angle of the preset model according to the feature information to obtain an adjusted preset model, it is further configured to generate a decoration component matching the target part according to the adjusted preset model; translate the decoration component to the position of the target part in the image so that the decoration component matches the target part.
[0122] An embodiment of the present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored on a memory through the data interface and executes part or all of the steps of any method recorded in the above method embodiments.
[0123] An embodiment of the present application further provides a computer storage medium. The computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments.
[0124] An embodiment of the present application further provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes part or all of the steps of any method recorded in the above method embodiments.
[0125] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above various methods. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.
[0126] The above-disclosed are only some embodiments of the present invention. Of course, the scope of the rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.
Claims
1. An image processing method, characterized in that, The method includes: Selecting the ear part in the image in the form of a target box to obtain the target box; Obtaining the vertex coordinates of the target box; Determining the position of the ear part according to the vertex coordinates of the target box, and calculating the aspect ratio of the target box according to the vertex coordinates of the target box; Calculating the aspect ratios of the ear model corresponding to the ear part at different tilt angles respectively; If the aspect ratio of the ear model matches the aspect ratio of the target box when the tilt angle of the ear model is the first angle value, then taking the first angle value as the tilt angle of the ear part, and taking the tilt angle and the position as the feature information of the ear part; Adjusting the size and tilt angle of the ear model according to the feature information to obtain an adjusted ear model, wherein the adjusted ear model is used to generate a decorative component matching the ear part.
2. The method according to claim 1, wherein Before adjusting the size and tilt angle of the ear model according to the feature information, the method further includes: Performing key point annotation on the ear model to obtain a plurality of key points; Determining the minimum bounding rectangle of the ear model, and obtaining the initial coordinates of the four vertices of the minimum bounding rectangle; Calculating the coordinates of the center point of the ear model according to the initial coordinates of the four vertices; Establishing a coordinate system with the center point as the coordinate origin; Calculating the target coordinates of each of the key points of the ear model in the coordinate system; Adjusting the size and tilt angle of the ear model according to the feature information includes: Adjusting the target coordinates of each of the key points of the ear model in the coordinate system according to the feature information to adjust the size and tilt angle of the ear model.
3. The method according to claim 2, wherein Calculating the target coordinates of each of the key points of the ear model in the coordinate system includes: Obtaining the initial coordinates of each key point among the plurality of key points; Calculating the target coordinates of each key point according to the initial coordinates of each key point and the coordinates of the center point.
4. The method according to claim 1, wherein Adjusting the size and tilt angle of the ear model according to the feature information to obtain an adjusted ear model includes: Rotating the ear model according to the tilt angle of the ear part so that the tilt angle of the rotated ear model matches the tilt angle of the ear part; Determining the size of the ear part according to the vertex position of the ear part; Scaling the size of the ear model according to the size of the ear part so that the size of the scaled ear model matches the size of the ear part.
5. The method according to claim 4, characterized in that, Scaling the size of the ear model according to the size of the ear part includes: Obtaining the target box corresponding to the ear part, and obtaining the width value and the height value of the ear part; Obtaining the width value and the height value of the ear model; Calculating the width ratio between the ear part and the ear model and the height ratio between the ear part and the ear model respectively; Determining the larger value among the width ratio and the height ratio, and then scaling the size of the ear model according to the larger value.
6. The method according to claim 4, wherein After adjusting the size and tilt angle of the human ear model according to the feature information to obtain an adjusted human ear model, the method further includes: Generating a decorative component that matches the ear part according to the adjusted human ear model; Translating the decorative component to the position of the ear part in the image so that the decorative component matches the ear part.
7. The method according to any one of claims 1-6, characterized in that, After adjusting the size and tilt angle of the human ear model according to the feature information to obtain an adjusted human ear model, it further includes: Generating a decorative component that matches the ear part according to the adjusted human ear model.
8. An image processing apparatus, characterized in that, The device includes an acquisition unit and a processing unit, where: The acquisition unit is configured to frame the ear part in the image in the form of a target box to obtain a target box; acquire the vertex coordinates of the target box; determine the position of the ear part according to the vertex coordinates of the target box, and calculate the aspect ratio of the target box according to the vertex coordinates of the target box; calculate the aspect ratios of the human ear model at different tilt angles respectively, and the shape structure of the human ear model matches the shape structure of the ear part; if the aspect ratio of the human ear model matches the aspect ratio of the target box when the tilt angle of the human ear model is the first angle value, then use the first angle value as the tilt angle of the ear part, and use the tilt angle and the position as the feature information of the ear part; The processing unit is further configured to adjust the size and tilt angle of the human ear model according to the feature information to obtain an adjusted human ear model, where the adjusted human ear model is used to generate a decorative component that matches the ear part.
9. A computer-readable storage medium, characterized in that, A computer program for electronic data interchange is stored, where the computer program causes a computer to execute the method according to any one of claims 1-7.