Image processing method and device, storage medium and equipment

By calculating and calibrating image distortion angles in the child companion robot camera, the problem of image distortion of the ground walking robot is solved, and the accuracy of user identity recognition and interactive experience are improved.

CN118968448APending Publication Date: 2024-11-15HEFEI IFLYTEK TOYCLOUD TECH
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410982035.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Because the child companion robot with ground walking function is short, the camera needs to raise up to shoot when taking the user's head image, resulting in image distortion, affecting the recognition accuracy, and making it difficult to accurately determine the user's identity.

Method used

The target image is obtained through the robot camera, the distortion angle is calculated, the image is calibrated, and the rectangular image is obtained, and then the pre-stored image is matched to determine the user's identity.

Benefits of technology

Improve the interactive experience between users and robots and ensure the accuracy of identity classification results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968448A_ABST
    Figure CN118968448A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method and device, a storage medium and equipment, and the method comprises the steps: firstly obtaining a target image containing the face information of a target user through a camera disposed on a robot, and then calculating a distortion angle of the target image through the shooting parameters of the camera; then, according to the distortion angle of the target image, the target image is calibrated, and the calibrated target image is obtained; and matching the calibrated target image with a pre-stored image, and determining an identification result of the target user according to a matching result. Visibly, according to the method and the device, the distortion angle of the target image is calculated by using the shooting parameters of the camera on the robot, and then the target image containing the face information of the target user is calibrated by using the distortion angle, so that the identity classification result of the target user can be more accurately determined based on the calibrated target image; and the interaction experience of the target user and the robot is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to an image processing method, apparatus, storage medium and device. Background Art

[0002] With the advancement of robotics technology, the types of robots are becoming more and more diverse, and their functions are becoming more and more powerful. Among them, the use of interactive robots is becoming more and more extensive. Through interaction with robots, people's lives can be greatly facilitated and comfortable.

[0003] Take the child companion robot as an example. It is a robot designed for children and integrates voice interaction, learning companionship, habit formation, audio-visual entertainment, etc. Among them, there is a type of robot with ground walking function, which can move freely indoors, bringing a new experience to children's companionship. However, the height of this type of child companion robot with ground walking function is usually shorter, about 30cm or less. When the child and the ground walking robot are on the ground, due to the height difference between the child and the robot, the robot needs to raise the camera to capture the user's head image. The image captured by this shooting method often has a large distortion, which affects the accuracy of image recognition, making it difficult to determine the user's identity classification result. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide an image processing method, apparatus, storage medium and equipment, which can effectively calibrate the distorted image captured by the robot camera, so that the user's identity classification result can be accurately determined based on the calibrated image, thereby improving the user's interaction experience with the robot.

[0005] The embodiment of the present application provides an image processing method, which is applied to a robot, wherein the robot is equipped with a camera, and includes:

[0006] Acquire a target image containing facial information of a target user through the camera;

[0007] Calculating the distortion angle of the target image using the shooting parameters of the camera;

[0008] Calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image;

[0009] The calibrated target image is matched with the pre-stored image, and the recognition result of the target user is determined according to the matching result.

[0010] In a possible implementation, before acquiring the target image including the target user's facial information through the camera, the method further includes:

[0011] Receive a preset start instruction sent by the target user, and control the robot to enter a user identification mode according to the preset start instruction.

[0012] In a possible implementation, after receiving a preset start instruction issued by the target user and controlling the robot to enter a user identification mode according to the preset start instruction, the method further includes:

[0013] In response to a preset start instruction issued by the target user, the location information of the target user is determined.

[0014] In a possible implementation, obtaining a target image including facial information of a target user through the camera includes:

[0015] According to the position information of the target user and the current position information of the robot, the robot is controlled to rotate so that the face of the target user appears in the shooting range of the camera and the distance between the robot and the target user is within a preset range;

[0016] The camera is used to capture a target image containing the target user's facial information.

[0017] In a possible implementation, calculating the distortion angle of the target image by using the shooting parameters of the camera includes:

[0018] According to the pitch angle of the camera and the distance between the robot and the target user, the angle between the center line of the camera and the horizontal plane is determined as the distortion angle of the target image.

[0019] In a possible implementation, the target image is an image showing the target user as a trapezoid, obtained by shooting from an upward angle or a downward angle with a camera; and calibrating the target image according to the distortion angle of the target image to obtain the calibrated target image includes:

[0020] According to the distortion angle of the target image, the short side of the target image presented as a trapezoid is compared with the long side, and the image is stretched to both sides in the same proportion, so as to obtain a target image that presents the target user as a rectangle, as a calibrated target image;

[0021] Alternatively, according to the distortion angle of the target image, the long side of the target image presented as a trapezoid is compared with the short side, and the target image presented as a trapezoid is compressed from both sides to the middle in the same proportion, so as to obtain a target image presenting the target user as a rectangle, which is used as the calibrated target image;

[0022] Alternatively, according to the distortion angle of the target image, the short sides of the target image that appears as a trapezoid are stretched half to both sides and the long sides are compressed half from both sides to the middle, so as to obtain a target image that presents the target user as a rectangle as the calibrated target image.

[0023] In a possible implementation, matching the calibrated target image with a pre-stored image and determining the recognition result of the target user according to the matching result includes:

[0024] The similarity between the calibrated target image and the pre-stored image is calculated, and the identity recognition result of the target user is determined according to the similarity calculation result.

[0025] The embodiment of the present application further provides an image processing device, which is applied to a robot, wherein the robot is equipped with a camera, and includes:

[0026] An acquisition unit, used for acquiring a target image including facial information of a target user through the camera;

[0027] A calculation unit, used to calculate the distortion angle of the target image using the shooting parameters of the camera;

[0028] A calibration unit, configured to calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image;

[0029] The matching unit is used to match the calibrated target image with the pre-stored image, and determine the recognition result of the target user according to the matching result.

[0030] In a possible implementation manner, the device further includes:

[0031] The control unit is used to receive a preset start instruction issued by the target user, and control the robot to enter a user identification mode according to the preset start instruction.

[0032] In a possible implementation manner, the device further includes:

[0033] The determination unit is used to determine the location information of the target user in response to a preset start instruction issued by the target user.

[0034] In a possible implementation, the acquiring unit includes:

[0035] a control subunit, configured to control the robot to rotate according to the position information of the target user and the current position information of the robot, so that the face of the target user appears in the shooting range of the camera and the distance between the robot and the target user is within a preset range;

[0036] The shooting subunit is used to use the camera to shoot a target image containing the facial information of the target user.

[0037] In a possible implementation, the computing unit is specifically configured to:

[0038] According to the pitch angle of the camera and the distance between the robot and the target user, the angle between the center line of the camera and the horizontal plane is determined as the distortion angle of the target image.

[0039] In a possible implementation, the target image is an image showing the target user as a trapezoid obtained by shooting from an upward angle or a downward angle with a camera; and the calibration unit is specifically configured to:

[0040] According to the distortion angle of the target image, the short side of the target image presented as a trapezoid is compared with the long side, and the image is stretched to both sides in the same proportion, so as to obtain a target image that presents the target user as a rectangle, as a calibrated target image;

[0041] Alternatively, according to the distortion angle of the target image, the long side of the target image presented as a trapezoid is compared with the short side, and the target image presented as a trapezoid is compressed from both sides to the middle in the same proportion, so as to obtain a target image presenting the target user as a rectangle, which is used as the calibrated target image;

[0042] Alternatively, according to the distortion angle of the target image, the short sides of the target image that appears as a trapezoid are stretched half to both sides and the long sides are compressed half from both sides to the middle, so as to obtain a target image that presents the target user as a rectangle as the calibrated target image.

[0043] In a possible implementation, the matching unit is specifically configured to:

[0044] The similarity between the calibrated target image and the pre-stored image is calculated, and the identity recognition result of the target user is determined according to the similarity calculation result.

[0045] The embodiment of the present application also provides an image processing device, including: a processor, a memory, and a system bus;

[0046] The processor and the memory are connected via the system bus;

[0047] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned image processing method.

[0048] An embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned image processing method.

[0049] The embodiment of the present application further provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned image processing method.

[0050] The embodiment of the present application provides an image processing method, device, storage medium and equipment, which first obtains a target image containing the target user's facial information through a camera configured on a robot, and then uses the camera's shooting parameters to calculate the distortion angle of the target image; then, according to the distortion angle of the target image, the target image is calibrated to obtain a calibrated target image; then, the calibrated target image can be matched with the pre-stored image, and the recognition result of the target user can be determined based on the matching result. It can be seen that since the embodiment of the present application first uses the shooting parameters of the camera on the robot to calculate the distortion angle of the target image, and then uses the distortion angle to calibrate the target image containing the target user's facial information, it can more accurately determine the identity classification result of the target user based on the calibrated target image, thereby improving the interactive experience between the target user and the robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of a scenario of interaction between a robot and a target user provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a target image showing a target user as a regular trapezoid and a corresponding calibrated target image obtained by shooting with a camera tilted upwards according to an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a target image showing a target user as an inverted trapezoid obtained by shooting from a camera looking down, provided in an embodiment of the present application;

[0056] Figure 5 One of the comparison example images of the target image and the calibrated target image provided in the embodiment of the present application;

[0057] Figure 6 The second comparison example diagram of the target image and the calibrated target image provided in the embodiment of the present application;

[0058] Figure 7 A schematic diagram of the optical path of camera imaging provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the near and far position relationship between the camera and the target user provided in the embodiment of the present application;

[0060] Fig. 9 A schematic diagram of the coordinates of each point before and after calibration of the target image provided in the embodiment of the present application;

[0061] Fig.10 A schematic diagram of the composition of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] With the rapid development of robotics technology, various types of robots are being applied to more and more fields, enabling people to use these robots to combine learning, work, entertainment, etc. to improve people's living standards.

[0063] Take the child companion robot as an example. It is a robot designed for children and integrates voice interaction, learning companionship, habit formation, audio-visual entertainment, etc. Among them, there is a type of robot with ground walking function. This type of robot can move freely indoors, bringing a new experience to children's companionship. In actual applications, robots often need to identify the identity of the user first in order to provide more appropriate personalized services, such as identifying whether the user is a pre-set child who needs companionship. If so, it can provide the child with corresponding companionship services through intelligent interaction with the child. However, if the user is identified as a stranger, it may be necessary to issue an alarm or other processing. Existing methods for child companion robots to identify user identities generally include the following two:

[0064] The first is a method of identifying user identity based on voiceprint. This identification method collects the user's voice in advance, extracts the user's voiceprint features from it, and then compares them with the voiceprint features in a pre-stored voiceprint database. The user's identity (such as whether it is a child who needs companionship or a stranger, etc.) can be identified based on the comparison results (such as whether there are consistent voiceprint features in the voiceprint database, or voiceprint features with a similarity higher than a certain value).

[0065] The second is a method of identifying user identity based on images. This identification method is to pre-collect the user's head image and then compare it with the pre-stored avatar information. The user's identity (such as whether it is a child who needs companionship or a stranger, etc.) can be identified based on the comparison results (such as whether they are consistent or the similarity is higher than a certain value).

[0066] Among them, when recognizing the user's identity based on images, for a child companion robot with the function of walking on the ground, its height is usually relatively short, about less than 30 cm. Therefore, when the child and the ground walking robot are on the ground, due to the height difference between the child and the robot, the robot needs to raise the camera to shoot in order to collect the user's head image. The image obtained by this shooting method often has a large distortion, which reduces the accuracy of image recognition, making it difficult to determine the user's identity classification result.

[0067] In order to solve the above defects, the present application provides an image processing method, firstly, a target image containing the target user's facial information is obtained through a camera configured on the robot, and then, the distortion angle of the target image is calculated using the shooting parameters of the camera; then, the target image is calibrated according to the distortion angle of the target image to obtain a calibrated target image; then, the calibrated target image can be matched with the pre-stored image, and the recognition result of the target user is determined according to the matching result. It can be seen that, since the embodiment of the present application first uses the shooting parameters of the camera on the robot to calculate the distortion angle of the target image, and then uses the distortion angle to calibrate the target image containing the target user's facial information, it is possible to more accurately determine the identity classification result of the target user based on the calibrated target image, thereby improving the interactive experience between the target user and the robot.

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] First embodiment

[0070] See also Figure 1 , is a flow chart of an image processing method provided in this embodiment, the method comprising the following steps:

[0071] S101: Acquire a target image including facial information of a target user through a camera.

[0072] In this embodiment, the image processing method provided by the present application is generally applied to any robot that needs to perform image processing. It should be noted that the present application does not limit the type of robot, for example, the robot may be a service robot or an operating robot, etc. The user interacting with it is defined as the target user, and the image containing the facial information of the target user obtained by the camera pre-configured on the robot is defined as the target image. Moreover, the present embodiment does not limit the type of the target image, for example, the target image may be a color image composed of the three primary colors of red (R), green (G), and blue (B), or a grayscale image, etc. Moreover, the present embodiment does not limit the resolution of the target image, for example, the target image may be a 720*480RGB image with a resolution, or may be a high-resolution 1920*1280RGB image, etc.

[0073] In addition, the present application does not limit the application scenarios of the robot. For example, the robot can be a child companion robot with the function of walking on the ground, or a robot serving in a restaurant, etc.

[0074] Figure 2 FIG. 1 is a schematic diagram showing a scenario in which a robot and a target user interact with each other in an embodiment of the present application. Figure 2 As shown, the robot and the target user are in the same room. The robot can be any home service robot, such as a child companion robot with ground walking function. The target user can be any child or other family member who needs companionship at home. It should be understood that Figure 2 The number and position of the robots and target users in the figure are only for illustration purposes. Any number of robots and target users may be provided as required.

[0075] The image processing method provided in the embodiment of the present application is generally executed by a controller in the robot, and accordingly, the image processing device can be set in the robot. However, it is easy for those skilled in the art to understand that the image processing method provided in the embodiment of the present application can also be executed by other terminal devices.

[0076] In order to more accurately determine the identity classification result of the target user and improve the interactive experience between the target user and the robot, firstly, a target image containing the target user's facial information needs to be acquired through a camera pre-configured on the robot to perform the subsequent step S102.

[0077] Specifically, an optional implementation method is that before obtaining the target image containing the target user's facial information through the camera, it is necessary to receive a preset start-up instruction issued by the target user, and control the robot to enter the user recognition interaction mode according to the preset start-up instruction, that is, wake up the robot according to the preset start-up instruction. Among them, the present application does not limit the specific content and wake-up method of the preset start-up instruction. For example, voice, gestures or other devices (such as watches, bracelets, etc.) can be used as the wake-up method of the preset start-up instruction to wake up the robot. After waking up the robot, the robot can be controlled to stop the current work, enter the user recognition mode, and wait for interaction with the target user to identify the user's identity classification result, so as to provide corresponding personalized services.

[0078] Further, after receiving the preset start instruction sent by the target user, the location information of the target user can be determined in response to the preset start instruction sent by the target user, which is used as an identification basis for determining the identity information of the target user.

[0079] Specifically, when the preset start-up instruction includes a preset voice start-up instruction, such as after the target user uses voice to wake up the robot, the location information of the target user can be located by using existing or future positioning methods combined with the microphone array to which the voice belongs.

[0080] On this basis, in order to more accurately determine the identity classification result of the target user and improve the interaction experience between the target user and the robot, it is further possible to obtain a target image containing the target user's facial information through a camera based on the target user's location information to execute subsequent steps S102 and S103.

[0081] Specifically, after determining the position information of the target user, if it is determined that the target user is not standing in front of the robot and does not appear in the robot's camera, in order to achieve effective interaction between the target user and the robot, the robot can be further controlled to rotate according to the position information of the target user and the current position information of the robot, so that the face of the target user can appear in the shooting range of the robot's camera. At the same time, in order to ensure the clarity of the target image, the depth map information obtained by the depth sensor can also be used to determine the human body depth information of the target user, and then the distance between the target user and the robot is determined. If it is determined that the distance between the target user and the robot is far, the robot can be controlled to rotate so that the distance between the robot and the target user is within a preset range, and then the robot's camera can be used to capture a target image containing the target user's facial information, and ensure that the captured target image is clear enough, so that the identity information of the target user can be more accurately identified through subsequent steps S102-S104.

[0082] Among them, the specific value of the preset range can be determined according to the actual situation and experience value, and the embodiment of the present application does not limit this. For example, the preset range can be set to within 3 meters.

[0083] S102: Calculate the distortion angle of the target image using the shooting parameters of the camera.

[0084] It should be noted that when the target image containing the target user's facial information is captured by the robot camera, since there is often a height difference between the robot (especially a child companion robot with ground walking function) and the target user, in order to capture the target image containing the target user's facial information, the robot needs to raise the camera to shoot or look down. The target image captured in this way often has a large distortion, which reduces the accuracy of target image recognition.

[0085] Therefore, in this embodiment, after the robot obtains the target image containing the target user's facial information using the camera through step S101, it can further use the shooting parameters of the camera when shooting the target image to calculate the distortion angle of the target image, so as to execute the subsequent step S103 to realize the calibration of the target image.

[0086] Among them, the shooting parameters of the camera refer to the parameters of each component involved in shooting the target image. And this application does not limit the specific content of the shooting parameters of the camera, which can be set according to actual conditions and experience values. For example, the shooting parameters of the camera may include but are not limited to the pitch angle range of the camera (the specific value is not limited, it can be 0 to 90°, or 0 to 180°, etc.), the pitch angle of the camera when shooting the target image, and the distance between the robot and the target user (measurable) and other data.

[0087] Specifically, an optional implementation method is to determine the angle between the center line of the camera and the horizontal plane according to the pitch angle when the camera captures the target image and the distance between the robot and the target user, as the distortion angle of the target image.

[0088] In this implementation, the distortion angle of the target image refers to the deformation angle of the target image containing the target user's facial information caused by the camera shooting angle problem. Figure 3 , which shows a schematic diagram of a target image showing a target user as a regular trapezoid obtained by shooting with a camera tilted upward and a corresponding calibrated target image, such as Figure 3As shown in (a), point O indicates the location of the camera installed on the robot, ∠AOB' indicates the pitch angle of the robot camera, EB indicates the body shape (height) of the target user, OF indicates the horizontal direction when the camera is in a horizontal state, OC is perpendicular to AF', and OF is perpendicular to EB. When using a camera to shoot, various parameters of the camera are usually adjusted to capture a clear target image. When the camera is adjusted as follows Figure 3 When shooting from an upward angle as shown in (a), a target image showing the target user as a regular trapezoid will be captured, such as Figure 3 As shown in (b), the ideal state is to adjust EB representing the target user's body shape (height) to AF' to ensure that the captured image will not be distorted. Figure 3 ∠ACE shown in (a) is the correction angle of the target image, and ∠COF represented by the black dot refers to the angle between the center line of the camera and the horizontal plane. Based on the relationship of equal angles ∠ACE=∠FCF'=∠COF, it can be determined that ∠COF is the measurable distortion angle of the target image, and it is also Figure 3 The trapezoidal correction angle of the target image shown in (b) represents the relationship between the target user (ie, EB representing the height) and the vertical plane angle of the camera normal.

[0089] Alternatively, see Figure 4 , which shows a schematic diagram of a target image of a target user presented as an inverted trapezoid obtained by shooting from a camera's top view, such as Figure 4 As shown in (a), point O still represents the location of the camera installed on the robot, ∠AOB' still represents the pitch angle of the robot camera, EB still represents the body shape (height) of the target user, OF still represents the horizontal direction when the camera is in a horizontal state, OC is still perpendicular to AF', and OF is still perpendicular to EB. When using a camera to shoot, various parameters of the camera are usually adjusted to capture a clear target image. When the camera is adjusted as follows Figure 4 When shooting from a bird's-eye view as shown in (a), a target image showing the target user as an inverted trapezoid will be captured, such as Figure 4 As shown in (b), the ideal state is to adjust EB representing the target user's body shape (height) to AF' to ensure that the captured image will not be distorted. Figure 4 In (a), ∠ACE is still the correction angle of the target image, and ∠COF represented by the black dot refers to the angle between the center line of the camera and the horizontal plane. It can also be determined that ∠COF is the measurable distortion angle of the target image, and it is still Figure 4 The trapezoidal correction angle of the target image shown in (b) represents the relationship between the target user (ie, EB representing the height) and the vertical plane angle of the camera normal.

[0090] S103: Calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image.

[0091] In this embodiment, after obtaining the target image containing the target user's facial information through step S101 and calculating the distortion angle of the target image through step S102, the target image can be further calibrated according to the distortion angle of the target image (the specific calibration method is not limited) to obtain a calibrated target image for executing the subsequent step S104.

[0092] Specifically, an optional implementation method is that after obtaining a target image containing the target user's facial information and calculating the distortion angle of the target image, the short side of the target image presented as a trapezoid is compared with the long side and stretched to both sides in proportion according to the distortion angle of the target image, so as to obtain a target image presenting the target user as a rectangle as the calibrated target image. Figure 3 Taking the ∠COF shown in (b) as an example, the distortion angle of the target image that can be measured is Figure 3 The short side of the target image shown in (b) is compared with the long side and stretched to both sides in the same proportion, so as to obtain a target image that presents the target user as a rectangle, which is used as the calibrated target image, as shown in FIG. Figure 3 As shown in the rectangle in the middle of (c).

[0093] Alternatively, after obtaining a target image containing the target user's facial information and calculating the distortion angle of the target image, the target image presented as a trapezoid may be further compressed from both sides to the middle by comparing the long side with the short side according to the distortion angle of the target image, so as to obtain a target image presenting the target user as a rectangle, as the calibrated target image. Figure 3 Taking the ∠COF shown in (b) as an example, the distortion angle of the target image that can be measured is Figure 3 The target image shown in (b) is compressed from both sides to the middle by comparing the long side with the short side, so as to obtain a target image that presents the target user as a rectangle, as shown in FIG. Figure 3 As shown in the rectangle at the top of (c).

[0094] Alternatively, after obtaining a target image containing the target user's facial information and calculating the distortion angle of the target image, the short sides of the target image that is presented as a trapezoid may be stretched half to both sides and the long sides may be compressed half from both sides to the middle according to the distortion angle of the target image, so as to obtain a target image that presents the target user as a rectangle, as the calibrated target image. Figure 3 Taking the ∠COF shown in (b) as an example, the distortion angle of the target image that can be measured is Figure 3The target image shown in (b) is stretched half the same as the short side to both sides and compressed half the long side from both sides to the middle to obtain a target image that presents the target user as a rectangle, as the calibrated target image, as shown in FIG. Figure 3 As shown in the rectangle at the bottom of (c).

[0095] For example: Assume that the target image captured by the camera at an angle of about 15° is as follows Figure 5 For the distorted image in the form of a regular trapezoid shown in (a), by executing the above steps S102-S103, the short sides of the target image are stretched half to both sides and the long sides are compressed half from both sides to the middle, so as to obtain a calibrated target image in the form of a rectangle, as shown in FIG. Figure 5 As shown in (b).

[0096] Alternatively, assuming that the target image obtained by tilting the camera upward and presenting the target user as a trapezoid is as follows Figure 6 For the distorted image shown in (a) (i.e., the target user is presented as "small on top and large on the bottom"), by performing the above steps S102-S103 to calibrate the target image, a calibrated target image in which the target user is presented as a rectangle (i.e., the target user is presented as a substantially normal proportion) can be obtained, as shown in FIG. Figure 6 As shown in (b).

[0097] It can be understood that, assuming that the camera is looking down to shoot, the target image that may be obtained in which the target user is presented as a trapezoid, that is, the target user is presented as "large on top and small on the bottom". By executing the above steps S102-S103 to calibrate the target image, a calibrated target image in which the target user is presented as a rectangle can also be obtained (that is, the target user is presented in a basically normal proportion). The specific implementation process and examples will not be repeated here.

[0098] Next, this embodiment will introduce a specific implementation process of "calibrating the target image according to the distortion angle of the target image to obtain a calibrated target image" in the above step S103, but does not limit the implementation process of step S103, and it can also be implemented in this way.

[0099] Specifically, first, Figure 7 In the camera imaging optical path diagram shown, u represents the object distance, v represents the image distance, f represents the focal length, H represents the object height, and h represents the image height. According to the imaging law of the convex lens, the following formula is established:

[0100]

[0101]

[0102] Combining the above formulas (1) and (2) gives the following formula (3):

[0103]

[0104] When the camera focal length f = 2.38 mm and the object distance u ranges from 20 mm to 24 mm, it can be obtained that f << u, and then the above formula (3) can be approximately converted into the following formula (4):

[0105]

[0106] Based on the above formula (4), it can be known that the image height h is inversely proportional to the object distance u.

[0107] Since the purpose of calibrating the trapezoidal target image is to restore the near-side imaging height to the far-side imaging height, it can be known through the above formula derivation that: calculating the ratio of the far-side imaging height to the near-side imaging height can be converted into calculating the ratio of the near-side object distance to the far-side object distance. See Figure 8 , which shows the near and far side position relationship between the camera and the target user, as Figure 8 shown, point O represents the position of the camera installed on the robot, OA represents the far-side object distance, OB represents the near-side object distance, OH represents the central object distance, AB represents the object front and back edge distance (i.e., the distance between the head and feet of the target user), OD represents the ground, θ represents the shooting angle (the angle between the camera normal and the plane where the target user's height is located, and the sum of it and the distortion angle is 90°), based on the above formula combined with Figure 8 the following formula can be obtained:

[0108] OB 2 = OD 2 + BD 2 (5)

[0109] OA 2 = OD 2 + AD 2 (6)

[0110] OD = OH * sinθ (7)

[0111] HD = OH * cosθ (8)

[0112]

[0113]

[0114] When AB, OH, and θ are known values, the value of OA / OB can be calculated from the above formulas, that is, the ratio of the far and near side object distances is calculated, which is also the ratio of the near-side image height to the far-side image height, so that the coordinates of each point before and after calibration can be calculated to achieve the trapezoidal calibration of the target image.

[0115] For example: Fig. 9 As shown, it shows that according to the distortion angle of the target image, the long side of the target image presented as a trapezoid is compared with the short side, and the target image is compressed from both sides to the middle in proportion to each other, so as to obtain a calibrated target image that presents the target user as a rectangle, wherein xFar represents the horizontal coordinate of the long side (i.e., the far side) in the target image before calibration, which can be selected according to the actual value of the target image; yFar1 represents the vertical coordinate of the upper edge of the long side (i.e., the far side) in the target image before calibration, and the coordinate value is used to obtain the coordinates of the four points, and the actual change in size does not affect the calibration result.

[0116] Assumptions Fig. 9 The upper left corner is the origin (0,0), the width and height of the target image are w320*h240 respectively, the right trapezoid side is the widest part of the image, that is, xnear=320, the shooting angle θ is 30°, the object distance (the distance OH from the camera to the target user) is 23, the height of the target user is 20, the value of xFar is 90, and the value of yFar1 is 10. Substituting these parameters into the above formulas (5)-(10), we can get:

[0117] OD=OH*sinθ=23*sin30=11.5;

[0118] HD=OH*cosθ=23*sqrt(3 / 2)≈19.9;

[0119] OB 2 =OD 2 +BD 2 =4.82;

[0120] OA 2 =OD 2 +AD 2 =41.5;

[0121] The ratio of the distance between the near and far edges: OA / OB = 0.116;

[0122] yFar2=h-yFar1, that is, 240-10=230;

[0123] xnear=w=320;

[0124] ynear1=h / 2-ratioDistace*(h / 2-yFar1)=107.24;

[0125] ynear2=h / 2+ratioDistace*(h / 2-yFar1)=132.76;

[0126] Based on this, we can calculate Fig. 9The values ​​of the 6 coordinates are as follows:

[0127] (xfar, yfar1) = (90,10), (xFar, yFar2) = (90,230), (xnear, ynear1) = (320,107.24), (x near, yfar1) = (320,10), (xnear, yfar2) = (320,230), (xnear, ynear2) = (320,132.76).

[0128] A calibrated target image presenting the target user as a rectangle can thus be determined based on these coordinates.

[0129] S104: matching the calibrated target image with the pre-stored image, and determining the recognition result of the target user according to the matching result.

[0130] In this embodiment, after obtaining the calibrated target image through step S103, the calibrated target image can be further matched with the pre-stored image, and the target user's identity recognition result can be determined based on the matching result, such as determining whether the target user is a child who needs to be accompanied, or other family members, or a stranger, etc. The specific method of matching the calibrated target image with the pre-stored image is not limited, and can be selected according to actual conditions and experience values. For example, the similarity between the calibrated target image and the pre-stored image can be calculated, and the target user's identity recognition result can be determined based on the similarity calculation result, so that according to the identity of the target user, a more suitable personalized service can be provided to the target user, thereby improving the interaction effect between the target user and the robot and the target user's interaction experience.

[0131] Among them, the pre-stored images refer to the head images of users of known identities pre-stored in the robot. For example, when the robot is a home service robot, the pre-stored images it stores may be the head images of children who need companionship and the head images of other family members (such as the child’s father, mother, grandfather, grandmother, etc.), which are used as the basis for identifying the target user’s identity information.

[0132] It should be noted that the present application does not limit the specific method for calculating the similarity between the calibrated target image and the pre-stored image, which can be selected according to actual conditions and experience. For example, the calibrated target image and the pre-stored image can be processed (i.e., encoded) using existing or future image feature extraction methods. For example, vgg16, resnet34, histogram of oriented gradients (Convolutional Neural Networks, HOG) feature extraction, scale-invariant feature transform (SIFT) feature extraction methods, etc. can be used to extract features that can characterize the image information of each from the calibrated target image and the pre-stored image. Here, they are respectively defined as the image features of the calibrated target image and the image features of the pre-stored image. The two image features should carry all the image information of their corresponding images, and the specific forms of expression of the two image features are not limited, and can be feature maps (vector matrices), etc., and then the similarity of the two image features is calculated. If the obtained similarity meets a preset threshold (the specific value is not limited, and can be set according to actual conditions and experience values, such as the preset threshold value can be set to 0.9, etc.), the identity of the target user can be determined as the user identity corresponding to the preset image, and corresponding personalized services can be provided for him.

[0133] It should be noted that the embodiments of the present application do not limit the method for extracting image features of the calibrated target image and the pre-stored image, nor do they limit the specific extraction process. An appropriate extraction method can be selected according to actual conditions, and corresponding feature extraction operations can be performed.

[0134] In summary, the image processing method provided in this embodiment first obtains a target image containing the target user's facial information through a camera configured on the robot, and then uses the camera's shooting parameters to calculate the distortion angle of the target image; then, according to the distortion angle of the target image, the target image is calibrated to obtain a calibrated target image; and then the calibrated target image can be matched with the pre-stored image, and the recognition result of the target user is determined based on the matching result. It can be seen that since the embodiment of the present application first uses the shooting parameters of the camera on the robot to calculate the distortion angle of the target image, and then uses the distortion angle to calibrate the target image containing the target user's facial information, it is possible to more accurately determine the identity classification result of the target user based on the calibrated target image, thereby improving the interactive experience between the target user and the robot.

[0135] Second embodiment

[0136] This embodiment will introduce an image processing device. For related content, please refer to the above method embodiment.

[0137] See also Fig.10 , is a schematic diagram of the composition of an image processing device provided in this embodiment, the device 1000 includes:

[0138] The acquisition unit 1001 is used to acquire a target image including facial information of a target user through the camera;

[0139] A calculation unit 1002 is used to calculate the distortion angle of the target image using the shooting parameters of the camera;

[0140] A calibration unit 1003 is used to calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image;

[0141] The matching unit 1004 is used to match the calibrated target image with the pre-stored image, and determine the recognition result of the target user according to the matching result.

[0142] In an implementation of this embodiment, the device further includes:

[0143] The control unit is used to receive a preset start instruction issued by the target user, and control the robot to enter a user identification mode according to the preset start instruction.

[0144] In an implementation of this embodiment, the device further includes:

[0145] The determination unit is used to determine the location information of the target user in response to a preset start instruction issued by the target user.

[0146] In an implementation of this embodiment, the acquiring unit 1001 includes:

[0147] a control subunit, configured to control the robot to rotate according to the position information of the target user and the current position information of the robot, so that the face of the target user appears in the shooting range of the camera and the distance between the robot and the target user is within a preset range;

[0148] The shooting subunit is used to use the camera to shoot a target image containing the facial information of the target user.

[0149] In an implementation of this embodiment, the calculation unit 1002 is specifically used for:

[0150] According to the pitch angle of the camera and the distance between the robot and the target user, the angle between the center line of the camera and the horizontal plane is determined as the distortion angle of the target image.

[0151] In an implementation of this embodiment, the target image is an image showing the target user as a trapezoid obtained by shooting from an upward angle or from a downward angle with a camera; the calibration unit 1003 is specifically configured to:

[0152] According to the distortion angle of the target image, the short side of the target image presented as a trapezoid is compared with the long side, and the image is stretched to both sides in the same proportion, so as to obtain a target image that presents the target user as a rectangle, as a calibrated target image;

[0153] Alternatively, according to the distortion angle of the target image, the long side of the target image presented as a trapezoid is compared with the short side, and the target image presented as a trapezoid is compressed from both sides to the middle in the same proportion, so as to obtain a target image presenting the target user as a rectangle, which is used as the calibrated target image;

[0154] Alternatively, according to the distortion angle of the target image, the short sides of the target image that appears as a trapezoid are stretched half to both sides and the long sides are compressed half from both sides to the middle, so as to obtain a target image that presents the target user as a rectangle as the calibrated target image.

[0155] In an implementation of this embodiment, the matching unit 1004 is specifically configured to:

[0156] The similarity between the calibrated target image and the pre-stored image is calculated, and the identity recognition result of the target user is determined according to the similarity calculation result.

[0157] Furthermore, an embodiment of the present application also provides an image processing device, including: a processor, a memory, and a system bus;

[0158] The processor and the memory are connected via the system bus;

[0159] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned image processing method.

[0160] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned image processing method.

[0161] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned image processing method.

[0162] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0163] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0164] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0165] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that: Applied to a robot, the robot is equipped with a camera, including: Acquire a target image containing facial information of a target user through the camera; Calculating the distortion angle of the target image using the shooting parameters of the camera; Calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image; The calibrated target image is matched with the pre-stored image, and the recognition result of the target user is determined according to the matching result.

2. The method according to claim 1, characterized in that Before acquiring the target image containing the target user's facial information through the camera, the method further includes: Receive a preset start instruction sent by the target user, and control the robot to enter a user identification mode according to the preset start instruction.

3. The method according to claim 2, characterized in that After receiving the preset start instruction issued by the target user and controlling the robot to enter the user identification mode according to the preset start instruction, the method further includes: In response to a preset start instruction issued by the target user, the location information of the target user is determined.

4. The method according to claim 3, characterized in that The step of obtaining a target image including facial information of a target user through the camera includes: According to the position information of the target user and the current position information of the robot, the robot is controlled to rotate so that the face of the target user appears in the shooting range of the camera and the distance between the robot and the target user is within a preset range; The camera is used to capture a target image containing the target user's facial information.

5. The method according to claim 1, characterized in that The step of calculating the distortion angle of the target image by using the shooting parameters of the camera includes: According to the pitch angle of the camera and the distance between the robot and the target user, the angle between the center line of the camera and the horizontal plane is determined as the distortion angle of the target image.

6. The method according to claim 1, characterized in that The target image is an image showing the target user as a trapezoid obtained by shooting from an upward angle or a downward angle with a camera; and calibrating the target image according to the distortion angle of the target image to obtain the calibrated target image includes: According to the distortion angle of the target image, the short side of the target image presented as a trapezoid is compared with the long side, and the image is stretched to both sides in the same proportion, so as to obtain a target image that presents the target user as a rectangle, as a calibrated target image; Alternatively, according to the distortion angle of the target image, the long side of the target image presented as a trapezoid is compared with the short side, and the target image presented as a trapezoid is compressed from both sides to the middle in the same proportion, so as to obtain a target image presenting the target user as a rectangle, which is used as the calibrated target image; Alternatively, according to the distortion angle of the target image, the short sides of the target image that appears as a trapezoid are stretched half to both sides and the long sides are compressed half from both sides to the middle, so as to obtain a target image that presents the target user as a rectangle as the calibrated target image.

7. The method according to any one of claims 1 to 6, characterized in that: The matching process of the calibrated target image with the pre-stored image and determining the recognition result of the target user according to the matching result includes: The similarity between the calibrated target image and the pre-stored image is calculated, and the identity recognition result of the target user is determined according to the similarity calculation result.

8. An image processing device, characterized in that: Applied to a robot, the robot is equipped with a camera, including: An acquisition unit, used for acquiring a target image including facial information of a target user through the camera; A calculation unit, used to calculate the distortion angle of the target image using the shooting parameters of the camera; A calibration unit, configured to calibrate the target image according to the distortion angle of the target image to obtain a calibrated target image; The matching unit is used to match the calibrated target image with the pre-stored image, and determine the recognition result of the target user according to the matching result.

9. An image processing device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Four-ocular video camera front face reconstruction method based on binocular stereo vision

    CN103065289A

  • Automatic keystone correction method and system based on projection television

    CN106101675A

  • Method and apparatus for identifying identity

    CN106446779A

  • Automatic identifying and shooting method and system based on intelligent robot

    CN107127758A

  • Method and device for controlling following behavior of robot, and robot

    CN107390721A