Image processing method and projection device

Abstract

The present disclosure provides an image processing method and a projection device. The image processing method includes: photographing a target object to obtain a first image and a second image, wherein the first image corresponds to a first part of the target object, the second image corresponds to a second part of the target object, and the first part and the second part are at least partially non-overlapping; obtaining a first feature point coordinate set and a second feature point coordinate set of the target object respectively according to the first image and the second image; obtaining coordinate system conversion information between the first image and the second image; according to the coordinate system conversion information, fusing the first feature point coordinate set with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object; according to the complete feature point coordinate set, obtaining a complete image of the target object. The present disclosure can increase the camera range, thereby meeting various application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of projection technology, and in particular, relates to an image processing method and a projection device. Background Art

[0002] Smart projector is one of the many intelligent electronic products in recent years. It is a projector that integrates functions such as human-computer interaction and multimedia interconnection. It can realize application services such as network search, network video, video on demand, digital music, online educational resources, and network video.

[0003] Smart projectors are generally equipped with a camera module for shooting, which is used to realize functions such as focus, image correction, and human-computer interaction. However, in different scenarios, the requirements for the shooting object, shooting angle, shooting accuracy, etc. are different. For example, the above-mentioned focus and image correction require shooting the projection screen, while human-computer interaction requires shooting the user, or there are other large-scale shooting requirements, which require not only shooting the user, but also shooting the scenery and other living / non-living things. The camera module currently equipped with smart projectors has a small field of view, which makes the shooting range too small to meet the current complex photography requirements. Summary of the invention

[0004] In order to solve the above problems and shortcomings of the prior art, the purpose of the present invention is to provide an image processing method and a projection device, which can increase the camera range and thus meet various application scenarios.

[0005] The present disclosure first provides an image processing method, the image processing method comprising:

[0006] Photographing a target object to obtain a first image and a second image, wherein the first image corresponds to a first portion of the target object, the second image corresponds to a second portion of the target object, and the first portion and the second portion at least partially do not overlap;

[0007] Acquire a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image respectively;

[0008] Acquire coordinate system conversion information between the first image and the second image;

[0009] According to the coordinate system conversion information, the first feature point coordinate set is merged with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object;

[0010] According to the complete feature point coordinate set, a complete image of the target object is obtained.

[0011] Optionally, the step of respectively acquiring a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image includes:

[0012] In response to the feature point coordinate set acquisition instruction, calling the human body feature recognition model;

[0013] The first image and the second image are input into a human feature recognition model to obtain a first feature point coordinate set and a second feature point coordinate set corresponding to the target object, respectively.

[0014] Optionally, the step of acquiring coordinate system conversion information between the first image and the second image includes:

[0015] Acquire a first coordinate system corresponding to the first image and a second coordinate system corresponding to the second image;

[0016] Selecting a first calibration point from the first coordinate system or the second coordinate system, mapping the first calibration point from one of the first coordinate system or the second coordinate system to the other, obtaining a second calibration point and a coordinate of the second calibration point based on its current coordinate system;

[0017] A transformation matrix between the first coordinate system and the second coordinate system is determined according to the coordinates of the first calibration point and the coordinates of the second calibration point, and the transformation matrix is ​​determined as coordinate system conversion information.

[0018] Optionally, the step of fusing the first feature point coordinate set with the second feature point coordinate set according to the coordinate system conversion information to obtain a complete feature point coordinate set of the target object includes:

[0019] According to the transformation matrix, the first feature point coordinate set is rotated and translated into the second coordinate system to obtain a third feature point coordinate set corresponding to the first feature point coordinate set in the second coordinate system; the third feature point coordinate set is superimposed and merged with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object;

[0020] Or, according to the transformation matrix, the second feature point coordinate set is rotated and translated into the first coordinate system to obtain a fourth feature point coordinate set corresponding to the second feature point coordinate set in the first coordinate system; the fourth feature point coordinate set is superimposed and fused with the first feature point coordinate set to obtain a complete feature point coordinate set of the target object.

[0021] Optionally, the step of fusing the first feature point coordinate set with the second feature point coordinate set according to the coordinate system conversion information to obtain a complete feature point coordinate set of the target object includes:

[0022] Superimposing the third feature point coordinate set with the second feature point coordinate set, or superimposing the fourth feature point coordinate set with the first feature point coordinate set, to obtain a feature point coordinate set of an overlapping portion and a feature point coordinate set of a non-overlapping portion;

[0023] Filter the feature point coordinate sets of the overlapping parts so that one of each pair of coincident feature point coordinates in the overlapping parts is retained;

[0024] The filtered feature point coordinate set of the overlapping parts is fused with the feature point coordinate set of the non-overlapping parts to obtain a complete feature point coordinate set of the target object.

[0025] Optionally, the step of filtering the feature point coordinate set of the overlapping parts so that one of each pair of coincident feature point coordinates in the overlapping parts is retained includes:

[0026] Obtain the confidence of each pair of coincident feature point coordinates in the overlapping part, which is generated when the human feature recognition model recognizes the feature points;

[0027] The one with relatively higher confidence in each pair of coincident feature point coordinates is taken as the final feature point coordinate.

[0028] The present disclosure also provides a projection device, including a projection device and a camera device for implementing the above-mentioned image processing method.

[0029] Optionally, the camera device includes a shell, and a first camera assembly arranged horizontally and a second camera assembly arranged vertically on the shell, the first camera assembly is used to capture the first image, and the second camera assembly is used to capture the second image.

[0030] Optionally, the second camera assembly and the shell are rotationally connected in a pitch direction to change the pitch angle of the second camera assembly; and / or, the first camera assembly and the shell are rotationally connected in a yaw direction to change the yaw angle of the first camera assembly.

[0031] Optionally, the camera device includes a shell, and a third camera assembly rotatably connected to the shell in a pitch direction and / or a yaw direction, and the third camera assembly is used to capture the first image and the second image.

[0032] Optionally, the first camera assembly includes a first color camera, a first infrared camera and a first infrared emitter which are arranged horizontally, and the first color camera is located between the first infrared camera and the first infrared emitter;

[0033] And / or, the second camera assembly includes a second color camera, a second infrared camera and a second infrared transmitter arranged vertically, and the second color camera is located between the second infrared camera and the second infrared transmitter.

[0034] Optionally, the second camera assembly is tilted relative to a horizontal plane to form an elevation angle, and a value range of the elevation angle is from 0 to 1 / 2 times of a field of view angle of the second camera assembly.

[0035] Compared with the prior art, the beneficial effects of the present disclosure include: firstly photographing the target object to obtain a first image and a second image, the first image corresponding to the first part of the target object, the second image corresponding to the second part of the target object, and the first part and the second part at least partially not overlapping; secondly, obtaining a first feature point coordinate set and a second feature point coordinate set of the target object respectively according to the first image and the second image; then obtaining the coordinate system conversion information between the first image and the second image; then, according to the coordinate system conversion information, merging the first feature point coordinate set and the second feature point coordinate set to obtain a complete feature point coordinate set of the target object; finally, according to the complete feature point coordinate set, obtaining a complete image of the target object. The present disclosure can overcome the defect in the prior art that the camera field angle is insufficient, resulting in a too small camera range. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solution of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 It is a schematic diagram of the camera range of the horizontal camera module;

[0038] Figure 2 is a schematic structural diagram of a camera device according to an embodiment of the present disclosure;

[0039] Figure 3 The steps of the image processing method of the present disclosure are shown in FIG. Figure 1 ;

[0040] Figure 4 It is a schematic diagram of the camera range of the vertical camera module;

[0041] Figure 5 The steps of the image processing method of the present disclosure are shown in FIG. Figure 2 ;

[0042] Figure 6 The steps of the image processing method of the present disclosure are shown in FIG. Figure 3 ;

[0043] Figure 7 is a schematic diagram of overlapping parts and overlapping part images according to an embodiment of the present disclosure;

[0044] Figure 8 is a schematic diagram of an image of a complete skeleton point coordinate set according to an embodiment of the present disclosure;

[0045] Fig. 9 The steps of the image processing method of the present disclosure are shown in FIG. Figure 4 ;

[0046] Fig.10 is a framework diagram of a processing unit of an embodiment of the present disclosure;

[0047] Fig.11 is a schematic structural diagram of a first camera assembly according to an embodiment of the present disclosure;

[0048] Fig.12 is a schematic structural diagram of a second camera assembly according to an embodiment of the present disclosure;

[0049] Fig.13 is a schematic diagram of the second camera assembly of the embodiment of the present disclosure Figure 1 ;

[0050] Fig.14 is a schematic diagram of the elevation angle of the second camera assembly in the embodiment of the present disclosure;

[0051] Fig.15 is a schematic diagram of a first camera assembly in a top-down direction of an embodiment of the present disclosure;

[0052] Fig.16 is a schematic diagram of the second camera assembly of the embodiment of the present disclosure Figure 2 ;

[0053] Fig.17 is a schematic structural diagram of a third camera assembly according to an embodiment of the present disclosure;

[0054] Fig.18 It is a framework diagram of an electronic device according to an embodiment of the present disclosure.

[0055] Description of reference numerals:

[0056] 1-first camera assembly; 2-second camera assembly; 3-device body;

[0057] 11-first color camera; 12-first infrared camera; 13-first infrared transmitter;

[0058] 21-second color camera; 22-second infrared camera; 23-second infrared transmitter;

[0059] 41-partial image acquisition module; 42-coordinate set acquisition module; 43-conversion information acquisition module; 44-coordinate set fusion module; 45-complete image acquisition module;

[0060] 5-third camera assembly; 51-third color camera; 52-third infrared camera; 53-third infrared transmitter. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0062] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0063] On the other hand, the prefixes “first”, “second”, etc. in this article are only used to distinguish different components in name, and do not represent or imply their importance, priority or sequence.

[0064] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0065] The camera on the projector can be used for human-computer interaction such as fitness and remote video chat, or it can simply meet common needs such as face recognition, taking pictures, and photography. These applications require a camera with a relatively large field of view. Taking the fitness scene as an example, in order to accurately capture the human body posture, a short-focus camera module is required to shoot the fitness person at a close distance. The field of view of the camera module is mostly wide horizontally and narrow vertically. Figure 1 As shown, the height of the image is smaller than the width, which results in the body of the fitness person not being fully captured.

[0066] It is understandable that, in addition to being applied to human-machine interaction scenarios such as fitness, the present disclosure can also be applied to ordinary shooting needs such as those mentioned above, and the shooting objects are not limited to humans, but also applicable to other animals, plants, or some inorganic objects, where animals can include invertebrates such as earthworms and insects such as bees, plants can include flowers, trees, etc., and inorganic objects can be figurines, plastic flowers, dummies, etc., which are not limited by the present disclosure. The present disclosure is described below using the fitness scenario as an example.

[0067] The present disclosure provides an image processing method. Figure 3 As shown, the image processing method includes:

[0068] Step 100, photograph the target object to obtain a first image and a second image, wherein the first image corresponds to a first part of the target object, and the second image corresponds to a second part of the target object, and the first part and the second part at least partially do not overlap. The first part and the second part may not overlap at all, for example, the first part is the upper body of the target object, and the second part is the lower body of the target object. The first part and the second part may not overlap partially, for example, the first part may be the torso and arms of the target object, and the second part may be the head, torso and legs of the target object. The first image and the second image have the same timestamp or the same time period.

[0069] In the above steps, the target object is a person because this is an embodiment of fitness. If it is to meet ordinary shooting needs, the target object can also be other animals, plants and inorganic objects mentioned above. The target object can be photographed simultaneously by the first camera component 1 and the second camera component 2 to obtain the first image and the second image respectively; in one embodiment, Figure 1 As shown, the width of the first image can be greater than the height, such as Figure 4 As shown, the height of the second image can be greater than the width, and the width of the first image is greater than the width of the second image, and the height of the second image is greater than the width of the first image. In other embodiments, the first image and the second image may only satisfy that the width of the first image is greater than the width of the second image, or the height of the second image is greater than the height of the first image.

[0070] The first camera assembly 1 captures a horizontal image, the second camera assembly 2 captures a vertical image, and the target object is a bodybuilder. For example, when the bodybuilder is in a standing position with his arms open and closed, the first camera assembly 1 can capture the bodybuilder's trunk and arms, but not the bodybuilder's head or feet; the second camera assembly 2 can capture the bodybuilder's trunk, head and feet, but not the outstretched arms.

[0071] In another embodiment, the target object can be photographed separately by a third camera assembly, which has rotational freedom in at least one axis and can track the target object. For example, when the target object is in a standing posture, the originally horizontal third camera assembly can quickly rotate to a vertical position after acquiring a first image of a portion of the torso and arms of the target object, so as to capture a second image of the complete torso (including the head, legs and middle torso) of the target object. Then, the first image and the second image can be processed to form a complete image of the target object. Of course, the third camera assembly can also be placed in a vertical position to acquire the second image first, and the above purpose can also be achieved by changing the order. For another example, when the target object is in a prone or lying state, if the field of view of the horizontally placed third camera assembly is not sufficient to support its shooting of the complete target object, a first image including the feet and lower torso of the target object can be first acquired, and then rotated in the yaw direction, and then a second image including the head and upper torso of the target object can be acquired, thereby processing to obtain a complete image of the target object, wherein, if the arms of the target object are in an extended state, and the first image and the second image cannot cover the arms, the third camera assembly can be rotated in the pitch direction at a suitable position to acquire a third image including the arms, and the third image can also be fused and processed to acquire a complete image. Of course, based on the embodiment of the lying state, if the field of view of the third camera assembly is large enough to directly acquire the first image of the rest of the body except the extended arms, it is also possible to only rotate in the pitch direction to acquire the second image including the arms.

[0072] Before shooting, the first camera assembly 1, the second camera assembly 2 and the third camera assembly need to be initialized so that the first camera assembly 1 and the second camera assembly 2 can accurately shoot the image of the human body posture.

[0073] Step 200, respectively obtain a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image. The feature points may be key bone points of the human body, such as the spine, knees, ankles, etc., or points on the human body contour, such as the top of the head, elbows, hips, armpits, etc.

[0074] In one embodiment, step 200 specifically includes:

[0075] First, in response to the feature point coordinate set acquisition instruction, the human body feature recognition model is called. The human body feature recognition model is a deep learning neural network, which can accurately identify key feature points in human body images after pre-training. The human body feature recognition model can be specifically a human skeleton recognition model.

[0076] Then the first image and the second image are input into the human feature recognition model to obtain the first feature point coordinate set and the second feature point coordinate set corresponding to the target object respectively. For example, by inputting the first image into the human skeleton recognition model, the feature points of the bodybuilder's shoulders, torso, elbows, hands, hips, and knees can be identified, and these feature points are assigned coordinate values. By inputting the second image into the human feature recognition model, the feature points of the bodybuilder's head, shoulders, torso, hips, knees, and feet can be identified, and these feature points are assigned coordinate values.

[0077] Step 300, obtaining coordinate system conversion information between the first image and the second image. The coordinate system conversion information can unify the first feature point coordinate set and the second feature point coordinate set into the same coordinate system. It is understandable that when only a single third camera assembly is used, since the posture of the third camera assembly when taking the first image and the second image is different (for example, the pitch angle and the yaw angle are inconsistent), the coordinate systems in which the first image and the second image are taken are also different, and it is also necessary to obtain the coordinate system conversion information.

[0078] In one embodiment, Figure 5 As shown, step 300 may specifically include the following steps:

[0079] Step 310 , obtaining a first coordinate system corresponding to the first image and a second coordinate system corresponding to the second image.

[0080] The above steps may include obtaining a first coordinate system corresponding to the first camera assembly 1 and a second coordinate system corresponding to the second camera assembly 2. The first coordinate system uses the optical center of the first camera assembly 1 as the coordinate origin, and the second coordinate system uses the optical center of the second camera assembly 2 as the coordinate origin. If there is a third image, a third coordinate system corresponding to the third image may also be obtained.

[0081] Step 320, select a first calibration point from the first coordinate system or the second coordinate system, map the first calibration point from one of the first coordinate system or the second coordinate system to the other, and obtain a second calibration point and the coordinates of the second calibration point based on its current coordinate system.

[0082] In one embodiment, step 320 may include: obtaining the coordinates of the first calibration point in the first coordinate system, and mapping the first calibration point to the second coordinate system, obtaining the second calibration point and its coordinates based on the second coordinate system. The first calibration point may include the coordinate origin of the first coordinate system and other feature points with known coordinates. If there is a third image, the third coordinate system corresponding to the third image may also be mapped to the first coordinate system or the second coordinate system.

[0083] In one embodiment, step 320 may include: obtaining the coordinates of the first calibration point in the second coordinate system, and mapping the first calibration point to the first coordinate system to obtain the second calibration point and its coordinates based on the first coordinate system. The first calibration point may include the coordinate origin of the second coordinate system and other feature points with known coordinates.

[0084] Step 330, according to the coordinates of the first calibration point and the coordinates of the second calibration point, determine the transformation matrix between the first coordinate system and the second coordinate system (that is, determine the transformation matrix of the first coordinate system to the second coordinate system, or the transformation matrix of the second coordinate system to the first coordinate system), and determine the transformation matrix as the coordinate system conversion information between the first camera component 1 and the second camera component 2. For example, to map the first calibration point to the second coordinate system, the first calibration point needs to be rotated and translated, so the difference between the coordinates of the first calibration point and the second calibration point can be represented by the amount of rotation and translation of the first calibration point through the transformation matrix.

[0085] The transformation matrix can be determined when the cameras of the first camera assembly 1 and the second camera assembly 2 are calibrated before use. The coordinates of the first calibration point are converted into a point cloud using the camera internal parameter data, and the ICP algorithm (based on the data registration method, using the nearest point search method to solve an algorithm based on free-form surfaces) is used to perform point cloud registration based on the second coordinate system. After registration, the transformation matrix of the first camera assembly 1 is obtained and saved, so that the three-dimensional coordinates of the first camera assembly 1 and the second camera assembly 2 are unified into the same world coordinate system.

[0086] In one embodiment, in order to expand the camera range, the second camera assembly 2 can preferably be tilted upward in the vertical direction, so that an initial rotation transformation amount can be given to the first calibration point in the point cloud registration stage, and further optimization can be performed to obtain a more accurate transformation matrix.

[0087] Step 400, according to the coordinate system conversion information, the first feature point coordinate set is merged with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object. For example, the first feature point coordinate set includes the feature point coordinates of the shoulders, torso, elbows, hands, hips, and knees, and the second feature point coordinate set includes the feature point coordinates of the head, shoulders, torso, hips, knees, feet, etc. The complete feature point coordinate set obtained after the fusion of the two can include the coordinate data of the feature points of the human head, shoulders, elbows, hands, chest, waist, hips, knees, feet, etc.

[0088] In one embodiment, Figure 6 As shown, step 400 may specifically include:

[0089] Step 410: Rotate and translate the first feature point coordinate set to the second coordinate system according to the transformation matrix to obtain a third feature point coordinate set corresponding to the first feature point coordinate set in the second coordinate system.

[0090] Step 420, superimpose and fuse the third feature point coordinate set with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object. For example, the third feature point coordinate set includes the feature point coordinates of the shoulders, torso, elbows, hands, hips, and knees, and the second feature point coordinate set includes the feature point coordinates of the head, shoulders, torso, hips, knees, feet, etc. The complete feature point coordinate set obtained after the fusion of the two may include the coordinate data of the feature points of the human head, shoulders, elbows, hands, chest, waist, hips, knees, feet, etc.

[0091] In one embodiment, step 420 may further include the following steps:

[0092] First, the third feature point coordinate set is superimposed on the second feature point coordinate set to obtain the feature point coordinate set of the overlapping part and the feature point coordinate set of the non-overlapping part. Figure 7 As shown, the feature point coordinate set of the overlapping parts includes the feature point coordinates of the shoulders, torso, hips and knees, and the feature point coordinate set of the non-overlapping parts includes the feature point coordinates of the head, hands and feet.

[0093] Then, the feature point coordinate set of the overlapping parts is filtered so that one of each pair of coincident feature point coordinates in the overlapping parts is retained. For example, in the overlapping parts, the shoulders, torso, hips and knees are coincident feature points, and the third feature point coordinate set contains the coordinates of the shoulder, torso, hips and knees, as well as the coordinates of the shoulder, torso, hips and knees in the second feature point coordinate set. Therefore, the shoulder, torso, hips and knees each contain a pair of coordinates. In the process of forming a complete feature point coordinate set, filtering is required, and only one of the feature point coordinates of the shoulder, torso, hips and knees is retained to avoid data duplication.

[0094] In one embodiment, the above steps may specifically include:

[0095] First, the confidence of each pair of coincident feature point coordinates in the overlapping part is obtained. The confidence is generated when the human feature recognition model recognizes the feature point. For example, when the feature point is a bone point, the human feature recognition model is a human skeleton recognition model. When the human skeleton recognition model recognizes the first image, the confidence of the knee feature point is 98.6%, that is, the accuracy of the knee feature point recognized at this time is 98.6%. When recognizing the second image, the execution degree of recognizing the knee feature point is 98.8%, that is, the accuracy of the knee feature point recognized at this time is 98.8%.

[0096] Then, the one with a relatively high confidence in each pair of overlapping feature point coordinates is used as the final feature point coordinate. For example, if the confidence of the knee feature point coordinate in the first feature point coordinate set is 98.6%, and the confidence of the knee feature point coordinate in the second feature point coordinate set is 98.8%, then the knee feature point coordinate in the second feature point coordinate set is retained. The final knee feature point coordinate is based on the knee feature point coordinate in the second feature point coordinate set.

[0097] Step 430, the filtered feature point coordinate set of the overlapping part is merged with the feature point coordinate set of the non-overlapping part to obtain a complete feature point coordinate set of the target object. Figure 8 As shown, the complete feature point coordinate set includes the coordinates of key feature points (such as the coordinates of the head, shoulders, torso, elbows, hands, hips, knees, feet, etc.), which can accurately reflect the human body posture.

[0098] In one embodiment, Fig. 9 As shown, step 400 may also specifically include:

[0099] Step 440: Rotate and translate the second feature point coordinate set to the first coordinate system according to the transformation matrix to obtain a fourth feature point coordinate set corresponding to the second feature point coordinate set in the first coordinate system.

[0100] Step 450, superimpose and fuse the fourth feature point coordinate set with the first feature point coordinate set to obtain a complete feature point coordinate set of the target object. For example, the first feature point coordinate set includes the feature point coordinates of the shoulders, torso, elbows, hands, hips, and knees, and the fourth feature point coordinate set includes the feature point coordinates of the head, shoulders, torso, hips, knees, feet, etc. The complete feature point coordinate set obtained after the fusion of the two may include the coordinate data of the feature points of the human head, shoulders, elbows, hands, chest, waist, hips, knees, feet, etc.

[0101] In one embodiment, step 450 may further include the following steps:

[0102] First, the fourth feature point coordinate set is superimposed on the first feature point coordinate set to obtain the feature point coordinate set of the overlapping part and the feature point coordinate set of the non-overlapping part. Figure 7 As shown, the feature point coordinate set of the overlapping parts includes the feature point coordinates of the shoulders, torso, hips and knees, and the feature point coordinate set of the non-overlapping parts includes the feature point coordinates of the head, hands and feet.

[0103] Then, the feature point coordinate set of the overlapping parts is filtered so that one of each pair of coincident feature point coordinates in the overlapping parts is retained. For example, in the overlapping parts, the shoulders, torso, hips and knees are coincident feature points, and the first feature point coordinate set contains the coordinates of the shoulder, torso, hips and knees, and the fourth feature point coordinate set contains the coordinates of the shoulder, torso, hips and knees. Therefore, the shoulder, torso, hips and knees all contain a pair of coordinates. In the process of forming a complete feature point coordinate set, filtering is required, and only one of the feature point coordinates of the shoulder, torso, hips and knees is retained to avoid data duplication.

[0104] In one embodiment, the above steps may specifically include:

[0105] First, the confidence of each pair of coincident feature point coordinates in the overlapping part is obtained. The confidence is generated when the human feature recognition model recognizes the feature point. For example, when the human feature recognition model recognizes the first image, the confidence of the knee feature point is 98.6%, that is, the accuracy of the knee feature point recognized at this time is 98.6%. When recognizing the second image, the execution degree of recognizing the knee feature point is 98.8%, that is, the accuracy of the knee feature point recognized at this time is 98.8%.

[0106] Then, the one with a relatively high confidence in each pair of overlapping feature point coordinates is used as the final feature point coordinate. For example, if the confidence of the knee feature point coordinate in the first feature point coordinate set is 98.6%, and the confidence of the knee feature point coordinate in the second feature point coordinate set is 98.8%, then the knee feature point coordinate in the second feature point coordinate set is retained. The final knee feature point coordinate is based on the knee feature point coordinate in the second feature point coordinate set.

[0107] Step 460, the filtered feature point coordinate set of the overlapping part is merged with the feature point coordinate set of the non-overlapping part to obtain a complete feature point coordinate set of the target object. Figure 8 As shown, the complete feature point coordinate set includes the coordinates of key feature points (such as the coordinates of the head, shoulders, torso, elbows, hands, hips, knees, feet, etc.), which can accurately reflect the human body posture.

[0108] Step 500: Obtain a complete image of the target object according to the complete feature point coordinate set, wherein the complete image may be a virtual image of the target object, such as a cartoon image or a simple line image.

[0109] In one embodiment, a true color image corresponding to the target object can also be generated based on the complete feature point coordinate set, the first image and the second image, which requires the processing unit to have higher performance. If a cartoon image is generated, the performance requirements of the processing unit are not high, and the calculation processing speed will also be faster.

[0110] In one embodiment, step 300 may further include the following steps:

[0111] First, the real-time elevation angle of the second camera assembly based on the horizontal plane is obtained. For example, according to the height of the target object, the elevation angle is adjusted so that the longitudinal field of view of the second camera assembly can just cover the longitudinal height of the target object. In this way, even if the longitudinal field of view of the second camera assembly is not large enough, the elevation angle can be adjusted to cover the height of the target object.

[0112] Then, according to the real-time elevation angle of the second camera assembly, the current coordinate system of the second camera assembly is determined. Because the elevation angle of the second camera assembly changes in real time, the coordinate system of the second camera assembly also changes. Therefore, it is necessary to update and determine the current coordinate system of the second camera assembly in real time.

[0113] Finally, coordinate system conversion information between the first camera assembly and the second camera assembly is determined according to the coordinate system of the first camera assembly and the current coordinate system of the second camera assembly, so that real-time coordinate system conversion information between the second camera assembly and the first camera assembly can be obtained.

[0114] The present disclosure also provides an image processing device, such as Fig.10 As shown, including:

[0115] The partial image acquisition module 41 is used to photograph a target object to obtain a first image and a second image, wherein the first image corresponds to a first part of the target object, and the second image corresponds to a second part of the target object, and the first part and the second part at least partially do not overlap;

[0116] A coordinate set acquisition module 42, configured to acquire a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image, respectively;

[0117] A conversion information acquisition module 43 is used to acquire coordinate system conversion information between the first image and the second image;

[0118] A coordinate set fusion module 44 is used to fuse the first feature point coordinate set with the second feature point coordinate set according to the coordinate system conversion information to obtain a complete feature point coordinate set of the target object;

[0119] The complete image acquisition module 45 acquires a complete image of the target object according to the complete feature point coordinate set.

[0120] The image processing device of this embodiment can adopt the image processing method provided by the above embodiment, firstly shoot the target object to obtain the first image and the second image, the first image corresponds to the first part of the target object, the second image corresponds to the second part of the target object, and the first part and the second part are at least partially non-overlapping; secondly, according to the first image and the second image, respectively obtain the first feature point coordinate set and the second feature point coordinate set of the target object; then obtain the coordinate system conversion information between the first image and the second image; then, according to the coordinate system conversion information, fuse the first feature point coordinate set with the second feature point coordinate set to obtain the complete feature point coordinate set of the target object; finally, according to the complete feature point coordinate set, obtain the complete image of the target object. The present disclosure can capture the complete standing, lying, side posture and other posture images of the fitness person at close range and in real time when the fitness person is doing fitness exercise.

[0121] The embodiment of the present disclosure also provides a projection device, including a projection device and a camera device for implementing the image processing method provided in the above embodiment. After the camera device captures the complete body posture image of the target object, it is projected and displayed through the projection device for reference by the target object.

[0122] In one embodiment, Figure 2 As shown, the camera device includes a shell 3, and a first camera assembly 1 arranged horizontally and a second camera assembly 2 arranged vertically on the shell, the first camera assembly 1 is used to shoot a first image, and the second camera assembly 2 is used to shoot a second image. Among them, the width of the first image can be greater than the height, the height of the second image can be greater than the width, and the width of the first image is greater than the width of the second image, and the height of the second image is greater than the width of the first image. In other embodiments, the first image and the second image may also only satisfy that the width of the first image is greater than the width of the second image, or the height of the second image is greater than the height of the first image. In this way, no matter the target object is in any body posture such as standing, lying prone or lying on its side, it can be completely photographed by the combination of the first camera assembly 1 and the second camera assembly 2, and finally projected and displayed by the projection device.

[0123] In one embodiment, the second camera assembly 2 and the housing 3 are connected to rotate in the pitch direction to change the pitch angle of the second camera assembly 2; and / or the first camera assembly 1 and the housing 3 are connected to rotate in the yaw direction to change the yaw angle of the first camera assembly 1. Fig.14 As shown in FIG. 1 , the pitch direction is the angle between the vertical line of the second camera assembly 2 and the horizontal plane. Fig.15As shown, in the yaw direction, the yaw angle of the first camera assembly 1 is angle b. Of course, in other embodiments, if the field of view of the first camera assembly 1 in the yaw direction is sufficient, it can also be configured to be fixed on the housing 3. Similarly, if the field of view of the second camera assembly 2 in the pitch direction is sufficient, it can also be configured to be fixed on the housing 3. Of course, if one is sufficient and the other is insufficient, the insufficient one can be configured to be rotatably connected to the housing 3. In the embodiments of the first camera assembly 1 and / or the second camera assembly 2 configured to be rotatably connected, when coordinate conversion is performed, the coordinate conversion information can be obtained by the mechanism driving the rotation thereof in conjunction with the controller. It can be understood that the coordinate conversion information between the second camera assembly 2 at different pitch angles and the first camera assembly 1 at different yaw angles is different, and in these embodiments, it may be necessary to obtain it in real time.

[0124] The second camera assembly 2 can adjust the pitch angle on the housing 3 according to the longitudinal height of the target object, i.e., the height, and can completely capture the body posture of the target object when the target object is in a standing posture. The first camera assembly 1 can rotate in the yaw direction on the housing 3 to adapt to the changes in the body posture of the target object such as lying flat, lying prone, and leaning sideways, and completely capture the body posture image.

[0125] In one embodiment, Fig.11 As shown, the first camera assembly 1 includes a first color camera 11, a first infrared camera 12 and a first infrared transmitter 13. The first color camera 11, the first infrared camera 12 and the first infrared transmitter 13 are located on the same horizontal line, and the first color camera 11 is located between the first infrared camera 12 and the first infrared transmitter 13. Among them, the first infrared transmitter 13 is used to emit infrared rays, which are reflected by the human body and received by the first infrared camera 12. The depth information of the infrared reflection point of the human body can be calculated through the time difference between the emission and reception of infrared rays. Then the first color camera 11 is used to capture the RGB image of the human body. The RGB image is combined with the depth information to obtain the three-dimensional coordinate information of the human body image.

[0126] In one embodiment, Fig.12As shown, the second camera assembly 2 includes a second color camera 21, a second infrared camera 22 and a second infrared transmitter 23. The second infrared camera 22 is located above the second infrared transmitter 23, and the second color camera 21 is located between the second infrared camera 22 and the second infrared transmitter 23. The second infrared transmitter 23 is used to transmit infrared rays, which are reflected by the human body and received by the second infrared camera 22. The depth information of the infrared reflection point of the human body can be calculated through the time difference between the infrared emission and reception. Then the second color camera 21 is used to capture the RGB image of the human body. The RGB image is combined with the depth information to obtain the three-dimensional coordinate information of the human body image.

[0127] In one embodiment, the first camera assembly 1 is located on the upper part of the device body 3, and the second camera assembly 2 is located below the first camera assembly 1, so that the field of view of the camera device of this embodiment can be increased.

[0128] In this embodiment, the first camera assembly 1 and the second camera assembly 2 may adopt a monocular structured light solution.

[0129] The field of view angle range of the first infrared camera 12 is: the horizontal viewing angle can be 50° to 70°, and in this embodiment, it can be selected as 58.4°; the vertical viewing angle is 40° to 60°, and in this embodiment, it can be selected as 45.5°.

[0130] The field of view angle range of the second infrared camera 22 is: the horizontal viewing angle can be 40° to 60°, and in this embodiment it can be selected as 45.5°; the vertical viewing angle can be 50° to 70°, and in this embodiment it is selected as 58.4°.

[0131] The first color camera 11 has a field of view angle range of 60° to 70° for the horizontal viewing angle, which can be 63.1° in this embodiment, and a vertical viewing angle of 45° to 60°, which can be 49.4° in this embodiment.

[0132] The field of view of the second color camera 21 has an angle range of 45° to 60° for the horizontal direction, which is 49.4° in this embodiment, and 60° to 70° for the vertical direction, which is 63.1° in this embodiment.

[0133] In one embodiment, Fig.13 and Fig.14As shown, the second camera assembly 2 is tilted relative to the horizontal plane to form an upward elevation angle a, and the value range of the elevation angle a is 0 to 1 / 2 times the field of view angle of the second camera assembly 2. Specifically, the maximum value of the elevation angle a can be 1 / 2 of the relatively smaller one of the horizontal viewing angle and the vertical viewing angle. For example, in this embodiment, the field of view angle of the second camera assembly 2 is 58.4° for the horizontal viewing angle and 45.5° for the vertical viewing angle, then the value range of the elevation angle a of the second camera assembly 2 is 0 to 22.75°. In this way, the camera range of the second camera assembly can be expanded to avoid the following problems: Fig.16 In the situation shown, the target object cannot be completely photographed. At the same time, the elevation angle a of the second camera assembly 2 is within a reasonable range to avoid excessive elevation angles, which in turn reduce the camera range.

[0134] In one embodiment, when the elevation angle a of the second camera assembly 2 is 20°, it is possible to capture a height range of 1975 mm at a distance of 1500 mm from the human body, that is, a fitness person with a height below 1975 mm can achieve full-body photography.

[0135] In one embodiment, when the height of the second camera assembly 2 is 200 mm, the elevation angle a is 20°, and the distance from the human body is 1975 mm, the camera height is 1971 mm; when the distance from the human body is 2475 mm, the camera height is 2554 mm.

[0136] In one embodiment, Fig.17As shown, the camera device may also include a housing 3, and a third camera assembly 5 rotatably connected to the housing in the pitch direction and / or the yaw direction, and the third camera assembly 5 is used to capture the first image and the second image. Specifically, the target object can be captured separately by the third camera assembly 5, and the third camera assembly 5 has a rotational degree of freedom in at least one of the pitch direction and the yaw direction, and it can track the target object. For example, when the target object is in a standing posture, the third camera assembly 5 originally placed horizontally can be quickly rotated to a vertical position after acquiring a first image of a part of the torso and arms of the target object, so as to capture a second image of the complete torso (including the head, legs and middle torso) of the target object, and then the first image and the second image can be processed to form a complete image of the target object. Of course, the third camera assembly 5 mentioned above can also be placed vertically to acquire the second image first, and the above purpose can also be achieved by changing the order. For another example, when the target object is in a prone or flat state, if the field of view of the horizontally placed third camera assembly 5 is not sufficient to support its shooting of the complete target object, the first image including the feet and lower torso of the target object can be first obtained, and then rotated in the yaw direction, and then the second image including the head and upper torso of the target object can be obtained, so as to obtain the complete image of the target object. If the arms of the target object are in an unfolded state, and the first image and the second image cannot cover the arms, the third camera assembly 5 can be rotated in the pitch direction at a suitable position to obtain the third image including the arms, and the third image can also be fused and processed to obtain the complete image. Of course, based on the embodiment of the flat state, if the field of view of the third camera assembly 5 is large enough to directly obtain the first image of the body other than the unfolded arms, it can also be rotated in the pitch direction to obtain the second image including the arms. The third camera assembly 5 may include a third color camera 51, a third infrared camera 52 and a third infrared transmitter 53. The third infrared transmitter 53 is used to transmit infrared rays, which are reflected by the human body and then received by the third infrared camera 52. The depth information of the infrared reflection point of the human body can be calculated by the time difference between the infrared emission and reception. Then the third color camera 51 is used to take the RGB image of the human body. The RGB image is combined with the depth information to obtain the three-dimensional coordinate information of the human body image.

[0137] This embodiment also provides an electronic device, such as Fig.18As shown, it includes a processor and a storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions (computer programs), or by controlling related hardware through instructions (computer programs), and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, a plurality of instructions are stored in the storage medium of the hardware device of the embodiment of the present disclosure, and the instructions can be loaded by the processor to execute the steps of any embodiment of the application startup method provided in the embodiment of the present disclosure.

[0138] The storage medium and the processor are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through a communication bus. The storage medium stores computer execution instructions for implementing the data access control method, including at least one software function module that can be stored in the storage medium in the form of software or firmware. The processor executes various functional applications and data processing by running the software program and module stored in the storage medium. The storage medium can be, but is not limited to, a random access storage medium (Random Access Memory, referred to as: RAM), a read-only storage medium (Read Only Memory, referred to as: ROM), a programmable read-only storage medium (Programmable Read-Only Memory, referred to as: PROM), an erasable read-only storage medium (Erasable Programmable Read-Only Memory, referred to as: EPROM), an electrically erasable read-only storage medium (Electric Erasable Programmable Read-Only Memory, referred to as: EEPROM), etc. Among them, the storage medium is used to store programs, and the processor executes the program after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned storage medium may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components. The processor may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic flow diagrams disclosed in this embodiment may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0139] The present disclosure also provides a computer-readable storage medium storing a computer program, which implements the above-mentioned application startup method when executed by a processor. The device where the storage medium is located can be an electronic device such as a smart projection device, a personal computer, a laptop computer, a smart phone, a tablet computer, etc.

[0140] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method is applicable to a projection device, wherein the projection device comprises a projection device and a camera device, wherein the camera device comprises a first camera component arranged horizontally and a second camera component arranged vertically, and the image processing method comprises: Shooting a target object to obtain a first image and a second image, wherein the first image corresponds to a first portion of the target object, the second image corresponds to a second portion of the target object, the first portion and the second portion at least partially do not overlap, the first image is shot by the first camera component, the second image is shot by the second camera component, the first image and the second image have the same timestamp or the same time period, the first image is a horizontal picture of the target object, and the second image is a vertical picture of the target object; Acquire a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image respectively; Acquire coordinate system conversion information between the first image and the second image; According to the coordinate system conversion information, the first feature point coordinate set is merged with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object; Obtaining a complete image of the target object according to the complete feature point coordinate set; The method further comprises: The complete image of the target object is displayed in real time through the projection device.

2. The image processing method according to claim 1, characterized in that: The step of respectively acquiring a first feature point coordinate set and a second feature point coordinate set of the target object according to the first image and the second image comprises: In response to the feature point coordinate set acquisition instruction, calling the human body feature recognition model; The first image and the second image are input into the human feature recognition model to obtain a first feature point coordinate set and a second feature point coordinate set corresponding to the target object, respectively.

3. The image processing method according to claim 1 or 2, characterized in that: The step of acquiring coordinate system conversion information between the first image and the second image comprises: Acquire a first coordinate system corresponding to the shooting of the first image, and a second coordinate system corresponding to the shooting of the second image; Selecting a first calibration point from the first coordinate system or the second coordinate system, mapping the first calibration point from one of the first coordinate system or the second coordinate system to the other, and obtaining a second calibration point and a coordinate of the second calibration point based on its current coordinate system; A transformation matrix between the first coordinate system and the second coordinate system is determined according to the coordinates of the first calibration point and the coordinates of the second calibration point, and the transformation matrix is ​​determined as the coordinate system conversion information.

4. The image processing method according to claim 3, characterized in that: The step of fusing the first feature point coordinate set with the second feature point coordinate set according to the coordinate system conversion information to obtain a complete feature point coordinate set of the target object includes: According to the transformation matrix, the first feature point coordinate set is rotated and translated into the second coordinate system to obtain a third feature point coordinate set corresponding to the first feature point coordinate set in the second coordinate system; the third feature point coordinate set is superimposed and merged with the second feature point coordinate set to obtain a complete feature point coordinate set of the target object; Or, according to the transformation matrix, the second feature point coordinate set is rotated and translated into the first coordinate system to obtain a fourth feature point coordinate set corresponding to the second feature point coordinate set in the first coordinate system; the fourth feature point coordinate set is superimposed and merged with the first feature point coordinate set to obtain a complete feature point coordinate set of the target object.

5. The image processing method according to claim 4, characterized in that: The step of fusing the first feature point coordinate set with the second feature point coordinate set according to the coordinate system conversion information to obtain a complete feature point coordinate set of the target object includes: Superimposing the third feature point coordinate set with the second feature point coordinate set, or superimposing the fourth feature point coordinate set with the first feature point coordinate set, to obtain a feature point coordinate set of an overlapping portion and a feature point coordinate set of a non-overlapping portion; Filtering the feature point coordinate set of the overlapping part so that one of each pair of coincident feature point coordinates in the overlapping part is retained; The filtered feature point coordinate set of the overlapping part is merged with the feature point coordinate set of the non-overlapping part to obtain a complete feature point coordinate set of the target object.

6. The image processing method according to claim 5, characterized in that: The step of filtering the feature point coordinate set of the overlapping part so that one of each pair of coincident feature point coordinates in the overlapping part is retained comprises: Obtaining the confidence of each pair of coincident feature point coordinates in the overlapping part, wherein the confidence is generated when the human feature recognition model recognizes the feature points; The one with a relatively higher confidence in each pair of coincident feature point coordinates is used as the final feature point coordinate.

7. A projection device, characterized in that: The invention comprises a projection device and a camera device for implementing the image processing method according to any one of claims 1 to 6.

8. The projection device according to claim 7, characterized in that: The camera device includes a shell, and a first camera component disposed horizontally and a second camera component disposed vertically on the shell. The first camera component is used to capture the first image, and the second camera component is used to capture the second image.

9. The projection device according to claim 8, characterized in that: The second camera assembly and the shell are rotationally connected in a pitch direction to change the pitch angle of the second camera assembly; and / or the first camera assembly and the shell are rotationally connected in a yaw direction to change the yaw angle of the first camera assembly.

10. The projection device according to claim 7, characterized in that: The camera device includes a shell, and a third camera assembly rotatably connected to the shell in a pitch direction and / or a yaw direction, and the third camera assembly is used to capture the first image and the second image.

11. The projection device according to claim 8, characterized in that: The first camera assembly includes a first color camera, a first infrared camera and a first infrared emitter which are arranged horizontally, and the first color camera is located between the first infrared camera and the first infrared emitter; And / or, the second camera assembly includes a second color camera, a second infrared camera and a second infrared emitter arranged vertically, and the second color camera is located between the second infrared camera and the second infrared emitter.

12. The projection device according to claim 8, characterized in that: The second camera assembly is tilted relative to a horizontal plane to form an elevation angle, and the elevation angle ranges from 0 to 1 / 2 times the field of view angle of the second camera assembly.

Images