Image processing method and apparatus, electronic device, and medium

Abstract

The application discloses an image processing method and device, electronic equipment and medium, and belongs to the field of electronic equipment. The method comprises the following steps: acquiring a to-be-processed image, determining a pre-matching region in the to-be-processed image according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image comprising a target object; determining an expansion region in the to-be-processed image according to the pre-matching region, wherein the pre-matching region is located in the expansion region; scaling the preset template to obtain at least two comparison templates with different sizes; calculating the first similarity of the images in different regions in the expansion region and each comparison template; and identifying the target object in the to-be-processed image according to the first similarity.

Description

Technical Field

[0001] This application belongs to the field of electronic equipment, and specifically relates to an image processing method, apparatus, electronic device, and medium. Background Technology

[0002] Image recognition technology is widely used in information gathering, healthcare, and security. By processing, analyzing, and understanding images, target objects within them can be identified, enabling further security checks, identity verification, distance measurement, and other functions.

[0003] The display effect of some target objects in the image varies greatly due to environmental factors such as season and weather, as well as the shooting parameters of the shooting module. For example, the sun, moon, clouds, and lights can display different colors and sizes due to environmental influences, resulting in low accuracy of the recognition results when using a fixed-size template for image recognition. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, electronic device, and medium that can solve the technical problem of low accuracy in identifying target objects that are greatly affected by environmental changes.

[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0006] The image to be processed is acquired, and a pre-matching region is determined in the image to be processed according to a preset template. The size of the pre-matching region is the same as the size of the preset template, which is an image that includes the target object.

[0007] The extended region is determined in the image to be processed based on the pre-matched region, and the pre-matched region is located within the extended region.

[0008] Scaling the preset template yields at least two comparison templates of different sizes;

[0009] Calculate the first similarity between the images located in different regions of the extended region and each comparison template;

[0010] The target object in the image to be processed is identified based on the first similarity score.

[0011] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire the image to be processed;

[0013] The first determining module is used to determine a pre-matching region in the image to be processed according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object;

[0014] The second determining module is used to determine the extended region in the image to be processed based on the pre-matching region, wherein the pre-matching region is located within the extended region;

[0015] The scaling module is used to scale the preset template to obtain at least two comparison templates of different sizes;

[0016] The first calculation module is used to calculate the first similarity between the images located in different regions of the extended region and each comparison template;

[0017] The recognition module is used to identify the target object in the image to be processed based on the first similarity.

[0018] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method provided in the first aspect.

[0019] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0020] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the method provided in the first aspect.

[0021] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as provided in the first aspect.

[0022] In this application, a pre-matching region is determined by using a preset template, and then an extended region is determined based on the pre-matching region. This can exclude other images in the image to be processed, except for the extended region, from participating in the calculation of the first similarity, thereby reducing the amount of computation and improving the efficiency of image processing. By scaling to obtain multiple comparison templates, and calculating the first similarity between the extended region and the comparison template, the recognition error caused by the different imaging size of the target object in different environments can be reduced or avoided, thereby improving the recognition accuracy. Attached Figure Description

[0023] Figure 1 This is one of the schematic flowcharts of an image processing method provided in an embodiment of this application;

[0024] Figure 2 This is one of the schematic diagrams of the image to be processed in the image processing method of this application embodiment;

[0025] Figure 3This is a second schematic diagram of the image to be processed in the image processing method of this application embodiment;

[0026] Figure 4 This is a second schematic flowchart of an image processing method provided in one embodiment of this application;

[0027] Figure 5 A first similarity curve is provided in one embodiment of this application;

[0028] Figure 6 This is the third schematic flowchart of an image processing method provided in one embodiment of this application;

[0029] Figure 7 This is the fourth schematic flowchart of an image processing method provided in one embodiment of this application;

[0030] Figure 8 This is the fifth schematic flowchart of an image processing method provided in one embodiment of this application;

[0031] Figure 9 This is the third schematic diagram of the image to be processed in the image processing method of this application embodiment;

[0032] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application;

[0033] Figure 11 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0034] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0037] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0038] Figure 1 This is a schematic flowchart of an image processing method provided in one embodiment of this application, as shown below. Figure 1 As shown in the figure, this application provides an image processing method, which may include:

[0039] S110, Obtain the image to be processed, and determine the pre-matching region in the image to be processed according to the preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object;

[0040] S120, determine the extended region in the image to be processed based on the pre-matched region, where the pre-matched region is located within the extended region;

[0041] S130, scale the preset template to obtain at least two comparison templates of different sizes;

[0042] S140, calculate the first similarity between the images located in different regions of the extended region and each comparison template;

[0043] S150, the target object in the image to be processed is identified based on the first similarity.

[0044] The image to be processed is an image captured by the camera module of an electronic device. The user can control the electronic device to capture a picture of the target object to obtain an image displaying the target object. When the identified target object is used for heading angle correction of the electronic device, the target object can be a landmark structure in the city where the electronic device is located, such as a tall building or a statue in a square. Of course, the target object can also be a celestial body or a combination of celestial bodies that are easy to capture. A single celestial body can be the sun or the moon, and a combination of celestial bodies can be the Big Dipper, Ursa Minor, Gemini, etc.

[0045] When the target object is a celestial body such as the sun or moon, the imaging effect of the target object in the image to be processed varies greatly under different environmental conditions. For example, the sun photographed at noon will appear smaller in the image to be processed compared to the sun photographed at sunset. Similarly, when the target object is a building, the imaging effect of the target object in the image to be processed also varies greatly under different environmental conditions. For example, if the building's decorative lights are on, the building photographed at night will appear larger in the image to be processed compared to the building photographed during the day.

[0046] The preset template is an image displaying the target object, which is pre-set by those skilled in the art as needed. The preset template can be compared with various regions of the object to be processed, and a second similarity score can be calculated between each region and the preset template. Pre-matching regions are then determined based on the second similarity score. The second similarity score can be grayscale similarity, hue similarity, saturation similarity, brightness similarity, etc. Optionally, the similarity score is grayscale similarity. It is understood that the larger the second similarity score, the greater the similarity between the region and the preset template, and the greater the likelihood that the region displays the target object. Therefore, the region corresponding to the maximum similarity score has the highest probability of displaying the target object. The region corresponding to the maximum similarity score can be set as the pre-matching region to facilitate subsequent image processing, excluding other regions in the image to be processed from participating in subsequent image processing, thereby reducing computational load and improving recognition efficiency.

[0047] To improve recognition accuracy, this application selects a larger extended region based on the pre-matching region and performs further similarity calculations on the images displayed in the extended region. The extended region can be a pre-set size area, sharing a common center point with the pre-matching region, forming an extended region extending outwards from the pre-matching region. Alternatively, the extended region and the pre-matching region can be set to share a common coordinate point, a common side edge, etc., as needed.

[0048] By scaling a preset template, you can obtain a smaller template and a larger template. Multiple comparison templates can be created by combining at least one smaller template and at least one larger template, and all comparison templates have different dimensions. Scaling and scaling can be based on the same rule or different rules. The preset template can be scaled according to a preset ratio, or according to a preset length or width.

[0049] The second similarity and the first similarity can be based on one or more parameters, such as both being grayscale value similarity, or they can be based on different parameters, such as: the first similarity is based on hue similarity, and the second similarity is based on grayscale value similarity. For example, the target object is the sun. See also... Figure 2Based on the preset template, a pre-matching region A1 with the same size as the preset template is identified in the captured image; then, an extended region A2 with a larger area is selected based on the pre-matching region A1, with the pre-matching region A1 within the extended region A2; the preset template is scaled to obtain a comparison template B1 with an enlarged size and a comparison template B2 with a reduced size; the first similarity is calculated based on the comparison template B1 and the comparison template B2 respectively; finally, the region corresponding to the maximum first similarity is determined as the region where the target object is located, and the object displayed in this region is the target object.

[0050] In this application, a pre-matching region is determined by using a preset template, and then an extended region is determined based on the pre-matching region. This can exclude other images in the image to be processed, except for the extended region, from participating in the calculation of the first similarity, thereby reducing the amount of computation and improving the efficiency of image processing. By scaling to obtain multiple comparison templates, and calculating the first similarity between the extended region and the comparison template, the recognition error caused by the different imaging size of the target object in different environments can be reduced or avoided, thereby improving the recognition accuracy.

[0051] In one embodiment, S110 includes:

[0052] S210, based on the first preset window frame, a sliding window frame sampling process is performed on the image to be processed to obtain multiple candidate images;

[0053] S220, Calculate the second similarity between the candidate image and the preset template;

[0054] S230, determine the display area of ​​the first candidate image in the image to be processed as the pre-matching area;

[0055] The size of the first preset window frame is the same as the size of the preset template, and the first candidate image is the image that corresponds to the maximum value of the second similarity among multiple candidate images.

[0056] Sliding window sampling is performed according to preset sliding rules, which specify the step size and direction of each slide of the first preset window frame. The first preset window frame selects a portion of the image to be processed. Each time the first preset window frame slides relative to the image to be processed, the image displayed corresponding to the first preset window frame changes, and the image displayed corresponding to the first preset window frame becomes a candidate image. Since the size of the first preset window frame is the same as the size of the preset template, the size of the pre-matching region is also the same as the size of the preset template. Because the imaging state of the target object varies under different seasons, climates, and other environments, the pre-matching region may only display a portion of the target object, or it may display other patterns besides the target object.

[0057] The image to be processed can be binarized first to obtain a black and white image corresponding to the image to be processed. Then, a first preset window frame with the same size as the preset template is set. The first preset window frame moves on the black and white image with a preset step size to select different images in the black and white image and compare them with the preset template to obtain a second similarity corresponding to different regions in the image to be processed.

[0058] Optionally, the formula for calculating the second similarity is as follows:

[0059]

[0060] Where T(x', y') is the grayscale value of a pixel in the preset template, and I(x, y) is the grayscale value of a pixel in the image to be processed.

[0061] Optionally, the formula for calculating the second similarity is as follows:

[0062]

[0063] Where T(x', y') is the grayscale value of a pixel in the preset template, I(x, y) is the grayscale value of a pixel in the image to be processed, W is the width of the preset template, and H is the height of the preset template.

[0064] Compared with the aforementioned similarity calculation formula, this calculation formula introduces the width value W and the height value H of the preset template, which can more accurately compare based on the center of the first preset window frame, which is beneficial for identifying centrally symmetrical target objects.

[0065] In one embodiment, S120 includes

[0066] Based on the first preset width, first preset height, second preset width, second preset height, pre-matching region, and preset selection rules, an expansion region is determined in the image to be processed.

[0067] Both the extended region and the pre-matching region include a first side, a second side, a third side, and a fourth side connected in sequence. The intersection of the first and second sides of the extended region is the first coordinate point, and the intersection of the third and fourth sides of the extended region is the second coordinate point. The intersection of the first and second sides of the pre-matching region is the third coordinate point, and the intersection of the third and fourth sides of the extended region is the fourth coordinate point. The preset selection rules include:

[0068] The x-coordinate of the first coordinate point is x1, which is obtained based on the first preset width and the x-coordinate of the third coordinate point.

[0069] The ordinate of the first coordinate point is y1, which is obtained based on the first preset height and the ordinate of the third coordinate point.

[0070] The x-coordinate of the second coordinate point is x2, which is obtained based on the second preset width and the x-coordinate of the fourth coordinate point.

[0071] The ordinate of the second coordinate point is y2, which is obtained based on the second preset height and the ordinate of the fourth coordinate point.

[0072] The first and second horizontal coordinates are different, and the first and second vertical coordinates are different.

[0073] The first side of the extended region is set parallel to the first side of the pre-matching region, the second side of the extended region is set parallel to the second side of the pre-matching region, the third side of the extended region is set parallel to the third side of the pre-matching region, and the fourth side of the extended region is set parallel to the fourth side of the pre-matching region.

[0074] Please see Figure 3 Optionally, the formulas for calculating the coordinates of the first and second coordinate points in the extended region are as follows:

[0075] bound_topleft x=max(0, local_topleft x-tmp_W / 2);

[0076] bound_topleft y=max(0, local_topleft y-tmp_H*1.5);

[0077] bound_bottomright x=min(bound_bottomright x+tmp_W*1.5, W);

[0078] bound_bottomright y=min(bound_bottomright y+tmp_H*1.5,H);

[0079] Where bound_topleft x is the x-coordinate of the top-left point P1 in the extended region, bound_topleft y is the x-coordinate of the top-left point P1 in the extended region, bound_bottomright x is the x-coordinate of the bottom-right point P2 in the extended region, and bound_bottomright x is the y-coordinate of the bottom-right point P2 in the extended region.

[0080] local_topleft x is the x-coordinate of the top-left coordinate point P3 of the pre-matching region, local_topleft y is the y-coordinate of the top-left coordinate point P3 of the pre-matching region, local_bottomright x is the x-coordinate of the bottom-right coordinate point P4 of the pre-matching region, and local_bottomright x is the y-coordinate of the bottom-right coordinate point P4 of the pre-matching region.

[0081] tmp_W is the width of the pre-matching region, and tmp_H is the height of the pre-matching region. tmp_W / 2 is the first preset width, tmp_H*1.5 is the first preset height, tmp_W*1.5 is the second preset width, tmp_H*1.5 is the second preset height, W is the width of the image to be processed, and H is the height of the image to be processed. Coordinate points P1 and P2 are the first and second coordinate points, respectively, and coordinate points P3 and P4 are the third and fourth coordinate points, respectively.

[0082] By setting the `max` function to take the maximum of two values ​​and the `min` function to take the minimum of two values, the expansion region is prevented from exceeding the image to be processed. Those skilled in the art will understand that the `max` function, `min` function, and the constants within these functions are all set based on the top-left vertex of the image to be processed being the origin of the coordinate system (0, 0). Those skilled in the art can refer to the above formulas to set the calculation formula for the coordinate points of the expansion region according to the actual situation.

[0083] Those skilled in the art will understand that, given the coordinates of the first and second coordinate points, a rectangular expansion region can be calculated based on these coordinates. In this embodiment, an expansion region is selected in the image to be processed using a pre-matched region and preset selection rules, thereby preventing the selected expansion region from exceeding the image to be processed.

[0084] In one embodiment, S130 includes:

[0085] Based on multiple preset scaling ratios, the preset templates are scaled proportionally to obtain at least two comparison templates of different sizes.

[0086] In other words, multiple preset scaling ratios can be pre-set as needed, thereby scaling the preset template proportionally according to the preset scaling ratios to obtain at least two comparison templates of different sizes. The multiple preset scaling ratios can be arithmetic sequences, geometric sequences, or random numbers. Optionally, those skilled in the art can pre-identify the target objects in multiple sample images to obtain the size of each target object, set the most frequently occurring size as the size of the preset template, and set the ratio of other frequently occurring sizes to the size of the preset template as the preset scaling ratio.

[0087] In one embodiment, S130 includes:

[0088] S131. Based on the preset minimum scaling ratio and the preset maximum scaling ratio, an arithmetic sequence with a preset tolerance is obtained. The arithmetic sequence includes multiple scaling ratios. The minimum value in the arithmetic sequence is the preset minimum scaling ratio, and the maximum value in the arithmetic sequence is the preset maximum scaling ratio.

[0089] S132, according to multiple scaling ratios in the arithmetic sequence, scale the preset template proportionally in sequence to obtain at least two comparison templates of different sizes.

[0090] Based on the preset minimum scaling ratio and the preset maximum scaling ratio, an arithmetic sequence of preset tolerances is obtained, which can quickly obtain multiple scaling ratios and scale the preset modules proportionally according to the scaling ratios.

[0091] In reference Figure 4 S130 includes:

[0092] S510: Based on multiple preset reduction ratios, proportionally reduce the preset templates to obtain multiple reduction comparison templates of different sizes;

[0093] S520: Based on the preset minimum magnification ratio and the preset maximum magnification ratio, an arithmetic sequence of preset tolerances is obtained. The arithmetic sequence includes multiple magnification ratios. The minimum value in the arithmetic sequence is the preset minimum magnification ratio, and the maximum value in the arithmetic sequence is the preset maximum magnification ratio.

[0094] S530, according to multiple magnification ratios in the arithmetic sequence, the preset template is magnified proportionally in sequence to obtain multiple magnified comparison templates of different sizes, including reduced comparison templates and magnified comparison templates.

[0095] For some target objects, the probability of image size reduction is less than the probability of image size expansion under different environmental conditions. Therefore, the number of reduction comparison templates can be set to be less than the number of magnification comparison templates as needed. Compared to setting multiple preset reduction ratios by listing them, the automatic generation of multiple magnification comparison templates of different sizes using preset tolerances via S520 and 530 can quickly obtain a larger number of magnification comparison templates.

[0096] For example, the preset template size is W*H, and two preset reduction ratios of 0.6 and 0.8 are set. A reduced comparison template with a size of 0.6W*0.6H and a reduced comparison template with a size of 0.8W*0.8H can be obtained through the preset reduction ratios.

[0097] Set the preset minimum magnification ratio P start Set the preset maximum magnification ratio P to 1.5. endThe value is set to 4, and the preset tolerance len is set to 0.5. An arithmetic sequence [1.5, 2, 2.5, 3, 3.5, 4] can be obtained. Based on the arithmetic sequence, the preset template S is enlarged proportionally to obtain enlarged comparison templates with dimensions of 1.5W*1.5H, 2W*2H, 2.5W*02.5H, 3W*3H, 3.5W*3.5H, and 4W*4H respectively.

[0098] The extended region can be sampled using a sliding window frame according to preset sliding rules. The preset sliding rules set the step size and direction of each slide of the second preset window frame. The second preset window frame selects a portion of the image in the extended region. Each time the second preset window frame slides, the image displayed corresponding to the second preset window frame changes. The image displayed corresponding to the second preset window frame is then compared with the comparison template to calculate multiple first similarities.

[0099] The method for calculating the first similarity is similar to that for the second similarity; both involve calculating the first similarity between the image corresponding to the preset window frame and the comparison template. The difference lies in the number of preset second windows. Each preset second window has the same size as a comparison template. Similarity is calculated only for the images displayed within the extended area using the preset second windows. Images outside the extended area in the image to be processed are not included in the first similarity calculation. The maximum value of the first similarity can be determined, and the image within the second preset window frame corresponding to this maximum value is the target object.

[0100] Please see Figure 5 The horizontal axis represents the different positions of the second preset window frames in the extended area, and the vertical axis represents the first similarity score. The maximum value of the first similarity score can be determined from this graph. The higher the first similarity score, the more similar the image displayed by the second preset window frame is to the comparison template, and the greater the probability that the image displayed by the second preset window frame is the target object.

[0101] Please see Figure 6 In one embodiment, the target object is the sun; S150 includes:

[0102] S710, Determine the target region in the extended region based on the first similarity;

[0103] S720 identifies the outline of the target region and calculates the outline area and outline perimeter based on the outline coordinates.

[0104] S730, calculates the roundness value of the target area based on the contour area and contour perimeter;

[0105] S740, if the roundness value is greater than the first preset threshold, determine the target object by displaying the image in the target area.

[0106] The roundness value of the target region can be calculated using its contour area and perimeter. Since the sun appears as a circle in the image to be processed, a higher roundness value indicates a greater likelihood that the target region contains the sun. Those skilled in the art can set a first preset threshold as needed. If the roundness value is greater than the first preset threshold, the matching region can be considered a target object in the image to be processed.

[0107] This application cleverly utilizes the shape of the sun to identify target objects in the image being processed based on similarity and roundness values, thereby improving recognition speed and reducing development difficulty.

[0108] Please see Figure 7 Identify the outline of the target region, including:

[0109] S810, extract the skeleton of the target object displayed within the target area;

[0110] S820, determine the minimum convex hull including the skeleton based on the skeleton, and determine the minimum convex hull as the contour of the target object.

[0111] In actual sun identification, the clouds surrounding the sun are very similar in color to the sun itself. To ensure the accuracy of sun detection, it is necessary to distinguish between the clouds and the sun. In this application, the target region is first transformed to obtain a grayscale image corresponding to the target region. Then, the grayscale image is binarized, and the skeleton of the binary image is extracted using a dilatational erosion morphological algorithm. Finally, the contour of the sun is extracted using the theory of minimum convex hull. Determining the skeleton first and then the minimum convex hull reduces the amount of computation and speeds up the contour extraction. The minimum convex hull is the smallest convex polygon that encloses all points; that is, the minimum convex hull can be considered equivalent to the contour of the target object.

[0112] Please see Figure 8 In one embodiment, after S150, the method further includes:

[0113] S910, Obtain the position information of the target object in the image to be processed;

[0114] S920, based on the position information, calculates the deviation information of the position information relative to the preset reference coordinates;

[0115] S930, calculate the first attitude information of the target object relative to the electronic device based on the deviation information;

[0116] S940: Acquire the second attitude information of the target object in the Earth coordinate system and the third attitude information of the electronic device in the Earth coordinate system, and correct the heading angle of the electronic device based on the first attitude information, the second attitude information and the third attitude information.

[0117] The image to be processed is an image captured by the camera module of an electronic device. The user can control the electronic device to capture images of target objects in its environment to obtain an image displaying the target object. The target object can be a landmark structure in the city where the electronic device is located, such as a tall building or a statue in a square. Alternatively, the target object can be a celestial body or a combination of celestial bodies that are easily captured. A single celestial body could be the sun or moon, and a combination of celestial bodies could be the Big Dipper, Ursa Minor, Gemini, etc. Those skilled in the art can identify the target object in the image to be processed using image recognition methods in related technologies. For example, the target object can be identified by comparing it with a preset image template, or by using a trained neural network model. Once the target object is identified in the image to be processed, its location information within the image can be obtained.

[0118] The first attitude information is the attitude information of the target object relative to the electronic device in a real environment. A more precise first attitude information is the attitude information of the target object relative to the camera module capturing the image to be processed in a real environment. This first attitude information may include roll angle and pitch angle. Those skilled in the art will understand that when a user holds the electronic device to capture the target object, rotating the electronic device about its vertical axis changes the roll angle of the target object relative to the electronic device, thus changing the movement of the target object in the image to be processed along a first direction; rotating the electronic device about its horizontal axis changes the pitch angle of the target object relative to the electronic device, thus changing the movement of the target object in the image to be processed along a second direction. The first and second directions are set perpendicularly. For example, consider a rectangular mobile phone. When the user holds the electronic device in the first position relative to a target object, the target object is centered in the image to be processed. The image captured by the electronic device is in portrait orientation. When the user rotates the electronic device around the vertical axis along direction A, the target object is to the right of the center of the image. When the user rotates the electronic device in the opposite direction of direction A, the target object is to the left of the center. When the user rotates the electronic device around the horizontal axis along direction B, the target object is above the center. When the user rotates the electronic device in the opposite direction of direction B, the target object is below the center. Therefore, the position information of the target object in the image is related to the relative posture information of the target object and the electronic device. Thus, the first posture information can be calculated using this position information.

[0119] The position information of the target object in the image to be processed can be represented as a combination of the horizontal coordinate (x) and the vertical coordinate (y). The preset reference coordinates are coordinates pre-defined by those skilled in the art. By comparing the position information with the preset reference coordinates, the deviation information from the preset reference coordinates can be obtained. The deviation information reflects the distance and direction of the target object's deviation from the preset reference coordinates in the image to be processed.

[0120] When the position information and preset reference coordinates are the same, the target object coincides with the preset reference coordinates. In this case, the normal of the camera module capturing the image to be processed, and the line connecting the target object and the Earth can be considered coaxial, and the heading angle of the electronic device and the direction angle of the target object can be considered the same. When the position information and preset reference coordinates are different, the position information of the target object in the image to be processed deviates from the preset reference coordinates. This indicates that the normal of the camera module capturing the image to be processed, and the line connecting the target object and the Earth, have an angle. The deviation information is related to the size of this angle. Therefore, the first attitude information of the target object relative to the electronic device can be calculated through the deviation information, so that the first attitude information can be obtained from the image analysis regardless of the target object's location in the image to be processed.

[0121] Second attitude information can be obtained by communicating with a preset server. When the target object is one or more celestial bodies, the second attitude information can include the elevation angle and azimuth angle of the target object in the Earth coordinate system. Both the elevation angle and azimuth angle are attitude angles in the Earth coordinate system. The elevation angle is the angle between the line connecting the celestial body and the location of the electronic device and the plane where the electronic device is located. The azimuth angle is the angle rotated clockwise from the north axis. Due to the relative motion between the celestial body and the Earth, the position of the target object in the Earth coordinate system changes over time. For example, when the celestial body is the Sun, due to the Earth's rotation and revolution, in a given latitude and longitude region, the elevation angle and azimuth angle are determined by the date of the year and the time of day. The formulas for calculating the solar elevation angle and azimuth angle are as follows:

[0122] θ s =asin(cosh*cosδ*cosφ+sinδ*sinφ)

[0123]

[0124] Where, θ S φ is the solar altitude angle. S φ is the solar azimuth angle, h is the hour angle, δ is the declination angle, and φ is the latitude.

[0125] Electronic devices can be equipped with IMU (Inertial Measurement Unit) sensors. These IMU sensors can detect third-attitude information, which includes the electronic device's heading angle γ, roll angle β, and pitch angle α in the Earth coordinate system. The heading angle γ is susceptible to interference from external magnetic and electric fields, so it needs to be corrected. Given the relative positions of the target object and the electronic terminal, the relative positions of the target object and the Earth, and the known roll angle β and pitch angle α of the electronic terminal relative to the Earth, the optimal heading angle can be calculated. This optimal heading angle is then used to correct the heading angle in the electronic terminal, enabling positioning and navigation functions.

[0126] In this embodiment, by identifying the target object, position information is obtained. Then, the first attitude information can be calculated based on the deviation information of the position information relative to the preset reference coordinates. The target object can be located at any position in the image to be processed, so as to reduce the requirements of the heading angle correction process on the image to be processed. The heading angle of the electronic device is corrected according to the first attitude information, the second attitude information, and the third attitude information, so that the heading angle of the electronic device can be corrected by taking an image of the image to be processed containing the target object. This avoids the inaccuracy of the heading angle caused by the magnetic field or electric field in the environment where the electronic device is located, and improves the accuracy of the heading angle stored in the electronic device.

[0127] In some embodiments, S120 includes:

[0128] S210, Based on the location information, determine the coordinates of the first center point of the target object to be displayed in the image to be processed;

[0129] S220, Based on the coordinates of the first center point and the preset reference coordinates, calculate the deviation information of the target object in the image to be processed relative to the center point of the image to be processed.

[0130] The target object displayed in the image to be processed can be represented as a combination of the horizontal coordinate (x) and the vertical coordinate (y). The position information can include the coordinates of each pixel of the target object. Those skilled in the art can determine the calculation method of the first center coordinates (x0, y0) based on the outline of the target object. For example, the coordinates of the first center point P1 are (x2 + (x1 - x2) / 2, y2 + (y1 - y2) / 2), where x1 is the maximum horizontal coordinate of the target object, x2 is the minimum horizontal coordinate of the target object, y1 is the maximum vertical coordinate of the target object, and y2 is the minimum vertical coordinate of the target object.

[0131] In the image to be processed, besides the pixels displaying the target object, other pixels can also be represented as a combination of horizontal (x) and vertical (y) coordinates. The pixel information of the image to be processed includes the number of horizontal pixels (w) measured along a first direction and the number of vertical pixels (h) measured along a second direction, where the first and second directions are perpendicular. The number of horizontal pixels (w) can be the maximum number of pixels along the first direction, and the number of vertical pixels (h) can be the maximum number of pixels along the second direction. Therefore, if the preset reference coordinates are the coordinates of the center point of the image to be processed, the preset reference coordinates are (w / 2, h / 2).

[0132] The deviation information may include a horizontal offset value and a vertical offset value. For example, the first center point coordinates (x0, y0) of the first center point P1 and the preset reference coordinates (x0', y0') of the preset reference coordinates P2, the horizontal offset value may be the difference between the horizontal coordinate x0' of the preset reference coordinates and the horizontal coordinate x0 of the first center point coordinates, and the vertical offset value may be the difference between the vertical coordinate y0' of the preset reference coordinates and the vertical coordinate y0 of the first center point coordinates.

[0133] In some embodiments, S130 includes:

[0134] S310, Obtain pixel information of the image to be processed, including the number of horizontal pixels and the number of vertical pixels;

[0135] S320: Calculate the first roll angle of the target object relative to the electronic device based on the lateral offset value, the preset lateral adjustment constant, and the number of lateral pixels.

[0136] S330, calculates the first pitch angle of the target object relative to the electronic device based on the vertical offset value, the preset vertical adjustment constant and the number of vertical pixels;

[0137] The deviation information includes lateral offset and longitudinal offset values, and the first attitude information includes the first roll angle of the target object relative to the electronic device and the first pitch angle of the target object relative to the electronic device.

[0138] The pixel information of the image to be processed includes the number of horizontal pixels measured along a first direction and the number of vertical pixels measured along a second direction, wherein the first and second directions are perpendicular. When the image to be processed is in portrait orientation, the first direction can be horizontal and the second direction can be vertical. The number of horizontal pixels can be the maximum size of the image to be processed measured along the horizontal direction, and the number of vertical pixels can be the maximum size of the image to be processed measured along the vertical direction. For example, when the image to be processed is in portrait orientation, the number of horizontal pixels *w* is the maximum size of the image in the width direction, and the number of vertical pixels *h* is the maximum size of the image in the length direction. For example, when the image to be processed is in landscape orientation, the number of horizontal pixels *w* is the maximum size of the image in the length direction, and the number of vertical pixels *w* is the maximum size of the image in the width direction. Those skilled in the art can determine whether the captured image is in portrait or landscape orientation based on the target object, background building, portrait, etc., in the image to be processed. For ease of description, the following embodiments are all described using an image to be processed in portrait orientation.

[0139] The horizontal coordinate x in the location information can be determined by the coordinate axis X extending along the first direction, and the vertical coordinate y can be determined by the coordinate axis Y extending along the second direction. Then the horizontal coordinate x is greater than or equal to 0 and less than or equal to the number of horizontal pixels w, and the vertical coordinate y is greater than or equal to 0 and less than or equal to the number of vertical pixels h.

[0140] The preset lateral adjustment constant and preset longitudinal adjustment constant are adjustment parameters preset by those skilled in the art, and these parameters are related to the parameters of the shooting module in the electronic device. Optionally, the first roll angle and the first pitch angle are calculated using the following formulas:

[0141]

[0142]

[0143] Among them, Roll x Pitch is the first roll angle of the target object relative to the electronic device. y I is the first pitch angle of the target object relative to the electronic device. center-x I is the x-coordinate of the preset reference coordinate system. center-y P is the ordinate of the preset reference coordinate system. center-x P represents the x-coordinate of the location information. center-y Roll represents the vertical coordinate of the location information. Base To preset the horizontal adjustment constant, Pitch Base To preset the longitudinal adjustment constant, I Width I represents the number of pixels horizontally. Height This represents the number of pixels vertically.

[0144] Optionally, the first roll angle and the first pitch angle can be calculated using the following formulas:

[0145]

[0146]

[0147] Among them, Roll x Pitch is the first roll angle of the target object relative to the electronic device. y I is the first pitch angle of the target object relative to the electronic device. center-x I is the x-coordinate of the preset reference coordinate system. center-y P is the ordinate of the preset reference coordinate system. center-x P represents the x-coordinate of the location information. center-y Roll represents the vertical coordinate of the location information. Base To preset the horizontal adjustment constant, Pitch Base To preset the longitudinal adjustment constant, I Width I represents the number of pixels horizontally. Height This represents the number of pixels vertically.

[0148] In some embodiments, the preset reference coordinates are the center point coordinates of the image to be processed; before S130, the following is included:

[0149] S410, when the target object is located in the first preset area of ​​the shooting preview image, records the first detected roll angle and the first detected pitch angle of the electronic device in the Earth coordinate system;

[0150] S420, when the target object is located in the second preset area of ​​the shooting preview image, records the second detection roll angle and the second detection pitch angle of the electronic device in the Earth coordinate system;

[0151] S430, a preset lateral adjustment constant is calculated based on the first detected roll angle and the second detected roll angle, and a preset longitudinal adjustment constant is calculated based on the first detected pitch angle and the second detected pitch angle;

[0152] The x-coordinate of the center point of the first preset area is different from that of the center point of the second preset area, and the y-coordinate of the center point of the first preset area is also different from that of the center point of the second preset area.

[0153] The horizontal and vertical coordinates of the center points of the first and second preset areas are different. This means that moving the target object from the first to the second preset area requires controlling the electronic device to deflect, causing changes in both its roll and pitch angles. Therefore, the first and second detected roll angles are inconsistent, as are the first and second detected pitch angles. Since the distance and direction of the target object's movement from the first to the second preset area are constant, the relationship constant between the target object's horizontal movement distance in the image and the change in the electronic device's roll angle can be calculated—this is the preset horizontal adjustment constant. Similarly, the relationship constant between the target object's vertical movement distance in the image and the change in the electronic device's pitch angle can also be calculated—this is the preset vertical adjustment constant.

[0154] like Figure 9 As shown, the dashed line indicates area A1, which is the upper left corner of the preview image and is the second preview area. The dashed line also indicates area A2, which is the center of the preview image and is the first preview area. The preset horizontal adjustment constant and preset vertical adjustment constant can be calculated using the following formulas:

[0155] Roll Base =Roll2-Roll1,

[0156] Pitch Base =Pitch2 - Pitch1,

[0157] Among them, Roll Base Preset horizontal adjustment constant and pitch Base As a preset longitudinal adjustment constant, Roll1 is the first detected roll angle, Pitch1 is the first detected pitch angle, Roll2 is the second detected roll angle, and Pitch2 is the second detected pitch angle.

[0158] Optionally, the image to be processed is a rectangular image. When the target object is located in the second preset area of ​​the captured preview image, the second preset area can be the upper left corner, lower left corner, lower right corner, and upper right corner of the captured preview image. Those skilled in the art can determine the addition and subtraction relationship in the above-mentioned preset horizontal adjustment constant and preset vertical adjustment constant calculation formula based on the specific position of the corner in the captured preview image.

[0159] S410 to S430 can calculate preset lateral and longitudinal adjustment constants using an electronic device, and then store these preset lateral and longitudinal adjustment constants in other devices of the same model for use. Once the preset lateral and longitudinal adjustment constants are set or calculated, they no longer need to be adjusted; they can be directly read and used during subsequent heading angle correction.

[0160] By measuring the roll and pitch angle offsets of the target object from the first preset area to the second preset area of ​​the preview image, preset lateral adjustment constants and preset longitudinal adjustment constants are calculated. Thus, in S240, the first roll angle and first pitch angle of the target object relative to the electronic device when capturing the image to be processed can be obtained based on pixel information, offset information, and preset lateral adjustment constants and preset longitudinal adjustment constants.

[0161] It can be displayed in the shooting preview image. Figure 9 The indicated prompt box (dashed box) instructs the user to move the electronic device so that the target object displayed in the shooting preview image is located within the prompt box, thereby placing the displayed target object within the first preset area or the second preset area of ​​the shooting preview image.

[0162] In some embodiments, S140 includes:

[0163] S510, based on the first attitude information, calculate the first coordinate vector of the target object relative to the electronic device;

[0164] S520: Acquire the second attitude information of the target object in the Earth coordinate system, and the third roll angle and third pitch angle of the electronic device in the Earth coordinate system;

[0165] S530, based on the second attitude information, calculate the second coordinate vector of the target object in the Earth coordinate system;

[0166] S540, based on the third roll angle, the third pitch angle and the first coordinate vector, construct a mapping function between the first coordinate vector and the transformed coordinate vector. The mapping function is a function of the heading angle variable, and the transformed coordinate vector is the vector of the first coordinate vector projected onto the Earth coordinate system.

[0167] S550 uses a mapping function to perform nonlinear optimization on the heading angle variable to obtain the constrained heading angle. The heading angle of the corrected electronic equipment is then used as the constrained heading angle.

[0168] The third attitude information includes the electronic equipment's third roll angle and third pitch angle in the Earth coordinate system; the angle between the transformed coordinate vector and the second coordinate vector is α, and when the heading angle variable is a constrained heading angle, the minimum value of the angle between the transformed coordinate vector and the second coordinate vector calculated by the mapping function is α.

[0169] As mentioned above, the first attitude information may include multiple attitude angles. In one embodiment, S510 involves vectorizing the coordinates of the target object relative to the electronic device using the first attitude angles. S510 includes:

[0170] Based on the first roll angle and the first pitch angle of the target object relative to the electronic device, the first coordinate vector of the target object relative to the electronic device is calculated. The first attitude information includes the first roll angle and the first pitch angle of the target object relative to the electronic device.

[0171] When the first attitude information includes the first roll angle and the first pitch angle, the first coordinate vector X1 is:

[0172] [cos(Pitch y )*cos(Roll x ),cos(Pitch y )*sin(Roll x ),sin(Pitch y )] T ,

[0173] Among them, Roll x For roll angle, Pitch y Let T be the first detected pitch angle, and T be the transpose matrix.

[0174] As mentioned above, the second attitude information may include multiple attitude angles. In one embodiment, S530 involves vectorizing the coordinates of the target object in the Earth coordinate system using the second attitude information. S530 includes:

[0175] Based on the elevation angle and azimuth angle of the target object relative to the Earth, the second coordinate vector of the target object relative to the electronic device is calculated. The second attitude information includes the elevation angle and azimuth angle of the target object relative to the Earth.

[0176] The second attitude information includes the target object's elevation angle and azimuth angle in the Earth coordinate system. The second coordinate vector X2 is:

[0177] [cos(θ s )*cos(φ s ),cos(θ s )*sin(φ s ),sin(θ s )]T ,

[0178] Where, θ S φ is the elevation angle of the target object. S Let T be the azimuth angle of the target object, and T be the transpose matrix.

[0179] As can be seen from the preceding text, the third attitude information may include the third heading angle γ, the third roll angle β, and the third pitch angle α in the Earth coordinate system acquired by the sensors of the electronic device. Since the purpose of this application is to correct the heading angle γ, that is, to assume that the accuracy of the third heading angle γ is low, the third heading angle γ obtained by the IMU sensor in the third attitude information may not be included in the calculation process of correcting the heading angle.

[0180] Similarly, the third attitude information can include multiple attitude angles, and S540 can include:

[0181] Based on the third roll angle and the third pitch angle, construct the rotation matrix of the electronic device in the Earth coordinate system with respect to the heading angle variable, and construct the mapping function of the first coordinate vector and the transformed coordinate vector based on the rotation matrix.

[0182] The rotation matrix Cnb is:

[0183]

[0184] Where γ is the heading angle, β is the third roll angle, and α is the third pitch angle.

[0185] In the coordinate system of the image to be processed captured by the electronic device, the line connecting the target object and the electronic device can be represented as the first coordinate vector X1. The heading angle γ in the rotation matrix Cnb can be set as a variable, resulting in a rotation matrix cmb(γ) with respect to the heading angle variable. The mapping function between the first coordinate vector and the transformed coordinate vector can then be expressed as cmb(γ)*X1. The heading angle variable can be any value A. cmb(A) can be obtained using this value A, along with the determined third roll angle and third pitch angle. Then, using the determined first coordinate vector, the transformed coordinate vector cmb(A)*X1, projected onto the Earth coordinate system, can be calculated. The transformed coordinate vector cmb*X1 is the theoretical coordinate vector of the target object relative to the electronic device in the Earth coordinate system, calculated from the image to be processed.

[0186] In the coordinate system of the electronic device, the line connecting the target object and the electronic device can be represented as the first coordinate vector X1. The transformed coordinate vector cmb*X1 represents the projection of the first coordinate vector X1 onto the Earth coordinate system through the cmb rotation matrix. That is, the transformed coordinate vector cmb*X1 is the theoretical coordinate vector of the target object in the Earth coordinate system calculated by the image to be processed and the variable heading angle γ.

[0187] In the Earth coordinate system, the line connecting the actual target object and the Earth can be understood as the second coordinate vector X2. When the angle α between the theoretical target object's transformed coordinate vector cmb*X1 in the Earth coordinate system and the actual target object's second coordinate vector X2 in the Earth coordinate system is 0, that is, when the transformed coordinate vector cmb*X1 and the second coordinate vector X2 are the same, the value of the heading angle variable is the actual heading angle. However, since the angle α between the transformed coordinate vector cmb*X1 and the second coordinate vector X2 may not be equal to 0 during the calculation process, nonlinear optimization is performed based on the mapping function to obtain the constrained heading angle. The constrained heading angle is a certain value of the heading angle variable, and when this heading angle variable is the heading angle variable, the nonlinear optimization condition is satisfied.

[0188] The specific conditions for nonlinear optimization are:

[0189] Argmin[α], 0≤γ≤360;

[0190] Where α is the angle between the transformed coordinate vector cmb(γ)*X1 and the second coordinate vector X2, and γ is the heading angle variable.

[0191] The minimum angle α between the transformed coordinate vector cmb*X1 and the second coordinate vector X2 is calculated using Argmin[α]. The smaller the angle α between the transformed coordinate vector cmb*X1 and the second coordinate vector X2, the closer the coordinate vector of the theoretical target object is to the coordinate vector of the real target object, and the closer the constrained heading angle of the theoretical target object's coordinate vector is to the real heading angle.

[0192] In one embodiment, the nonlinear optimization condition is:

[0193] argmax[cos(X2,cnb*X1)], 0≤γ≤360.

[0194] Those skilled in the art will understand that in trigonometric functions, the cosine value is 1 when the angle between the two sides is 0°, 1 when the angle between the two sides is 90°, and -1 when the angle between the two sides is 180°. 1 is the maximum cosine value. That is, the maximum cosine value of the angle α between the transformed coordinate vector cmb*X1 and the second coordinate vector X2 is less than or equal to 1. When cos(α) is at its maximum cosine value, the angle α between the transformed coordinate vector cmb*X1 and the second coordinate vector X2 is minimized. Therefore, it can be further deduced that the larger the cosine value of the transformed coordinate vector cmb*X1 and the second coordinate vector X2, the closer the theoretical target object is to the coordinates of the real target object, and the closer the constrained heading angle of the theoretical target object's coordinate vector is to the real heading angle.

[0195] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0196] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application, as shown below. Figure 10 As shown, the image processing apparatus may include:

[0197] The acquisition module 901 is used to acquire the image to be processed;

[0198] The first determining module 902 is used to determine a pre-matching region in the image to be processed according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object;

[0199] The second determining module 903 is used to determine the extended region in the image to be processed based on the pre-matching region, wherein the pre-matching region is located within the extended region;

[0200] The scaling module 904 is used to scale the preset template to obtain at least two comparison templates of different sizes;

[0201] The second calculation module 905 is also used to calculate the first similarity between the images located in different regions of the extended region and each comparison template;

[0202] The recognition module 906 is used to identify the target object in the image to be processed based on the first similarity.

[0203] In this application, a pre-matching region is determined by using a preset template, and then an extended region is determined based on the pre-matching region. This can exclude other images in the image to be processed, except for the extended region, from participating in the calculation of the first similarity, thereby reducing the amount of computation and improving the efficiency of image processing. By scaling to obtain multiple comparison templates, and calculating the first similarity between the extended region and the comparison template, the recognition error caused by the different imaging size of the target object in different environments can be reduced or avoided, thereby improving the recognition accuracy.

[0204] In an optional example, the second determining module 902 is further configured to:

[0205] Based on the first preset width, first preset height, second preset width, second preset height, pre-matching region, and preset selection rules, an expansion region is determined in the image to be processed.

[0206] Both the extended region and the pre-matching region include a first side, a second side, a third side, and a fourth side connected in sequence. The intersection of the first and second sides of the extended region is the first coordinate point, and the intersection of the third and fourth sides of the extended region is the second coordinate point. The intersection of the first and second sides of the pre-matching region is the third coordinate point, and the intersection of the third and fourth sides of the extended region is the fourth coordinate point. The preset selection rules include:

[0207] The x-coordinate of the first coordinate point is x1, which is obtained based on the first preset width and the x-coordinate of the third coordinate point.

[0208] The ordinate of the first coordinate point is y1, which is obtained based on the first preset height and the ordinate of the third coordinate point.

[0209] The x-coordinate of the second coordinate point is x2, which is obtained based on the second preset width and the x-coordinate of the fourth coordinate point.

[0210] The ordinate of the second coordinate point is y2, which is obtained based on the second preset height and the ordinate of the fourth coordinate point.

[0211] The first and second horizontal coordinates are different, and the first and second vertical coordinates are different.

[0212] In an optional example, the recognition module 906 further includes:

[0213] A determining unit is used to determine the target region in the extended region based on a first similarity.

[0214] The contour recognition unit is used to identify the contour of the target area and calculate the contour area and contour perimeter based on the contour coordinates.

[0215] The roundness value calculation unit is used to calculate the roundness value of the target area based on the contour area and contour perimeter.

[0216] The determining unit is also used to determine the image displayed in the target area as the target object when the roundness value is greater than a first preset threshold.

[0217] In an optional example, the image processing apparatus further includes a correction module 907, which includes:

[0218] The first acquisition unit is used to obtain the position information of the target object in the image to be processed;

[0219] The first calculation unit is used to calculate the deviation information of the position information relative to the preset reference coordinates based on the position information.

[0220] The second calculation unit is used to calculate the first attitude information of the target object relative to the electronic device based on the deviation information.

[0221] The second acquisition unit is used to acquire the second attitude information of the target object in the Earth coordinate system and the third attitude information of the electronic device in the Earth coordinate system.

[0222] The correction unit is used to correct the heading angle of the electronic device based on the first attitude information, the second attitude information, and the third attitude information.

[0223] By identifying the target object and obtaining its position information, the first attitude information can be calculated based on the deviation of the position information from the preset reference coordinates. The target object can be located at any position in the image to be processed, thereby reducing the requirements of the heading angle correction process on the image to be processed. The heading angle of the electronic device is corrected based on the first attitude information, the second attitude information, and the third attitude information, so that the heading angle of the electronic device can be corrected by taking an image of the image to be processed containing the target object. This avoids the inaccuracy of the heading angle caused by the magnetic or electric field in the environment where the electronic device is located, and improves the accuracy of the heading angle stored in the electronic device.

[0224] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, content-enabled computer, handheld computer, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0225] The image processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0226] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 9 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0227] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 100, including a processor 110 and a memory 119. The memory 119 stores a program or instructions that can run on the processor 110. When the program or instructions are executed by the processor 110, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0228] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0229] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0230] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 111, network module 112, audio output unit 113, input unit 114, sensor 115, display unit 116, user input unit 111, interface unit 118, memory 119, and processor 110.

[0231] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0232] The input unit 114 is used to acquire the image to be processed;

[0233] The processor 110 is used to determine a pre-matching region in the image to be processed according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object;

[0234] Processor 110 is also configured to determine an extended region in the image to be processed based on a pre-matched region, the pre-matched region being located within the extended region;

[0235] The processor 110 is also used to scale the preset template to obtain at least two comparison templates of different sizes;

[0236] Processor 110 is also used to calculate the first similarity between the images located in different regions of the extended region and each comparison template;

[0237] The processor 110 is also used to identify the target object in the image to be processed based on the first similarity.

[0238] In this application, a pre-matching region is determined by using a preset template, and then an extended region is determined based on the pre-matching region. This can exclude other images in the image to be processed, except for the extended region, from participating in the calculation of the first similarity, thereby reducing the amount of computation and improving the efficiency of image processing. By scaling to obtain multiple comparison templates, and calculating the first similarity between the extended region and the comparison template, the recognition error caused by the different imaging size of the target object in different environments can be reduced or avoided, thereby improving the recognition accuracy.

[0239] Optionally, the processor 110 is also used for:

[0240] Based on the first preset width, first preset height, second preset width, second preset height, pre-matching region, and preset selection rules, an expansion region is determined in the image to be processed.

[0241] Both the extended region and the pre-matching region include a first side, a second side, a third side, and a fourth side connected in sequence. The intersection of the first and second sides of the extended region is the first coordinate point, and the intersection of the third and fourth sides of the extended region is the second coordinate point. The intersection of the first and second sides of the pre-matching region is the third coordinate point, and the intersection of the third and fourth sides of the extended region is the fourth coordinate point. The preset selection rules include:

[0242] The x-coordinate of the first coordinate point is x1, which is obtained based on the first preset width and the x-coordinate of the third coordinate point.

[0243] The ordinate of the first coordinate point is y1, which is obtained based on the first preset height and the ordinate of the third coordinate point.

[0244] The x-coordinate of the second coordinate point is x2, which is obtained based on the second preset width and the x-coordinate of the fourth coordinate point.

[0245] The ordinate of the second coordinate point is y2, which is obtained based on the second preset height and the ordinate of the fourth coordinate point.

[0246] The first and second horizontal coordinates are different, and the first and second vertical coordinates are different.

[0247] Optionally, the processor 110 is also used for:

[0248] The target region is determined in the extended region based on the first similarity.

[0249] Identify the outline of the target region and calculate the outline area and perimeter based on the outline coordinates;

[0250] Calculate the roundness value of the target area based on the contour area and contour perimeter;

[0251] If the roundness value is greater than the first preset threshold, the image displayed in the target area is determined to be the target object.

[0252] Optionally, the processor 110 is also used for:

[0253] Obtain the location information of the target object in the image to be processed;

[0254] Based on the location information, calculate the deviation information of the location information relative to the preset reference coordinates;

[0255] Based on the deviation information, calculate the first attitude information of the target object relative to the electronic device;

[0256] Sensor 115 is used to acquire the second attitude information of the target object in the Earth coordinate system and the third attitude information of the electronic device in the Earth coordinate system;

[0257] The processor 110 is also used to correct the heading angle of the electronic device based on the first attitude information, the second attitude information, and the third attitude information.

[0258] By identifying the target object and obtaining its position information, the first attitude information can be calculated based on the deviation of the position information from the preset reference coordinates. The target object can be located at any position in the image to be processed, thereby reducing the requirements of the heading angle correction process on the image to be processed. The heading angle of the electronic device is corrected based on the first attitude information, the second attitude information, and the third attitude information, so that the heading angle of the electronic device can be corrected by taking an image of the image to be processed containing the target object. This avoids the inaccuracy of the heading angle caused by the magnetic or electric field in the environment where the electronic device is located, and improves the accuracy of the heading angle stored in the electronic device.

[0259] It should be understood that, in this embodiment, the input unit 114 may include a graphics processing unit (GPU) 1141 and a microphone 1142. The GPU 1141 processes the image data to be processed, which is a static image or video obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 116 may include a display panel 1161, which may be configured as a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 117 includes at least one of a touch panel 1171 and other input devices 1172. The touch panel 1171 is also called a touch screen. The touch panel 1171 may include a touch detection device and a touch controller. Other input devices 1172 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0260] The memory 119 can be used to store software programs and various data. The memory 119 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 119 may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 119 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0261] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0262] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0263] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0264] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0265] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0266] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0267] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0268] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0269] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain an image to be processed, and determine a pre-matching region in the image to be processed according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object; An extended region is determined in the image to be processed based on the pre-matched region, wherein the pre-matched region is located within the extended region; Scaling the preset template yields at least two comparison templates of different sizes; Calculate the first similarity between the images located in different regions of the extended region and each of the comparison templates; The target object in the image to be processed is identified based on the first similarity.

2. The method according to claim 1, characterized in that, Determining an extended region in the image to be processed based on the pre-matched region includes: Based on the first preset width, the first preset height, the second preset width, the second preset height, the pre-matching region, and the preset selection rules, the extended region is determined in the image to be processed. Both the extended region and the pre-matching region include a first side, a second side, a third side, and a fourth side connected in sequence. The intersection of the first and second sides of the extended region is a first coordinate point, the intersection of the third and fourth sides of the extended region is a second coordinate point, the intersection of the first and second sides of the pre-matching region is a third coordinate point, and the intersection of the third and fourth sides of the pre-matching region is a fourth coordinate point. The preset selection rules include: The x-coordinate of the first coordinate point is x1, which is obtained based on the first preset width and the x-coordinate of the third coordinate point; The ordinate of the first coordinate point is y1, which is obtained based on the first preset height and the ordinate of the third coordinate point; The x-coordinate of the second coordinate point is x2, which is obtained based on the second preset width and the x-coordinate of the fourth coordinate point; The ordinate of the second coordinate point is y2, which is obtained based on the second preset height and the ordinate of the fourth coordinate point; Wherein, the x-coordinate of the first coordinate point and the x-coordinate of the second coordinate point are different, and the y-coordinate of the first coordinate point and the y-coordinate of the second coordinate point are different.

3. The method according to claim 1, characterized in that, The target object is the sun; The target object in the image to be processed is obtained based on the first similarity recognition, including: The target region is determined in the extended region based on the first similarity. Identify the outline of the target region, and calculate the outline area and outline perimeter based on the coordinates of the outline; Calculate the roundness value of the target region based on the outline area and the outline perimeter; If the roundness value is greater than a first preset threshold, the image displayed in the target area is determined to be the target object.

4. The method according to claim 1, characterized in that, After identifying the target object in the image to be processed based on the first similarity, the process includes: Obtain the position information of the target object in the image to be processed; Based on the location information, calculate the deviation information of the location information relative to the preset reference coordinates; Based on the deviation information, calculate the first attitude information of the target object relative to the electronic device; The second attitude information of the target object in the Earth coordinate system and the third attitude information of the electronic device in the Earth coordinate system are obtained, and the heading angle of the electronic device is corrected according to the first attitude information, the second attitude information and the third attitude information.

5. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the image to be processed; The first determining module is used to determine a pre-matching region in the image to be processed according to a preset template, wherein the size of the pre-matching region is the same as the size of the preset template, and the preset template is an image including the target object; The second determining module is used to determine an extended region in the image to be processed based on the pre-matching region, wherein the pre-matching region is located within the extended region; A scaling module is used to scale the preset template to obtain at least two comparison templates of different sizes; The first calculation module is used to calculate the first similarity between the images located in different regions of the extended region and each of the comparison templates; The recognition module is used to identify the target object in the image to be processed based on the first similarity.

6. The image processing apparatus according to claim 5, characterized in that, The second determining module is further configured to determine the extended region in the image to be processed based on the first preset width, the first preset height, the second preset width, the second preset height, the pre-matching region, and the preset selection rules. Both the extended region and the pre-matching region include a first side, a second side, a third side, and a fourth side connected in sequence. The intersection of the first and second sides of the extended region is a first coordinate point, the intersection of the third and fourth sides of the extended region is a second coordinate point, the intersection of the first and second sides of the pre-matching region is a third coordinate point, and the intersection of the third and fourth sides of the pre-matching region is a fourth coordinate point. The preset selection rules include: The x-coordinate of the first coordinate point is x1, which is obtained based on the first preset width and the x-coordinate of the third coordinate point; The ordinate of the first coordinate point is y1, which is obtained based on the first preset height and the ordinate of the third coordinate point; The x-coordinate of the second coordinate point is x2, which is obtained based on the second preset width and the x-coordinate of the fourth coordinate point; The ordinate of the second coordinate point is y2, which is obtained based on the second preset height and the ordinate of the fourth coordinate point; Wherein, the x-coordinate of the first coordinate point and the x-coordinate of the second coordinate point are different, and the y-coordinate of the first coordinate point and the y-coordinate of the second coordinate point are different.

7. The image processing apparatus according to claim 5, characterized in that, The target object is the sun; the identification module includes: A determining unit is configured to determine a target region in the extended region based on the first similarity. A contour recognition unit is used to recognize the contour of the target region and calculate the contour area and contour perimeter based on the coordinates of the contour. A roundness value calculation unit is used to calculate the roundness value of the target region based on the contour area and the contour perimeter; The determining unit is further configured to determine, when the roundness value is greater than a first preset threshold, the image displayed in the target area as the target object.

8. The image processing apparatus according to claim 5, characterized in that, The image processing device further includes a correction module, the correction module comprising: The first acquisition unit is used to obtain the position information of the target object in the image to be processed; The first calculation unit is used to calculate the deviation information of the position information relative to the preset reference coordinates based on the position information; The second calculation unit is used to calculate the first attitude information of the target object relative to the electronic device based on the deviation information; The second acquisition unit is used to acquire the second attitude information of the target object in the Earth coordinate system and the third attitude information of the electronic device in the Earth coordinate system. The correction unit is used to correct the heading angle of the electronic device based on the first attitude information, the second attitude information, and the third attitude information.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1 to 4.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 4.

Images