Target detection method, terminal device, and medium

Abstract

The present disclosure provides a target detection method. The method comprises: acquiring a first scene image captured by a camera (101); acquiring current position and attitude information of the camera (102); adjusting the first scene image based on the current position and attitude information of the camera to obtain a second scene image (103); and performing target detection on the second scene image (104). Furthermore, the present disclosure also provides a terminal device and a medium.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of U.S. Patent Application Serial No. 62 / 947,314, filed December 12, 2019, in the U.S. Patent and Trademark Office, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of image recognition, and in particular, to a target detection method, a target detection apparatus, a terminal device, and a medium. BACKGROUND

[0004] A specific object, such as a face or a car, can be detected in an image by target detection. Target detection is widely used in the technical field of image recognition.

[0005] Currently, mainstream target detection methods divide the detection process into two stages. The first stage is to extract multiple regions (i.e., region proposals) that can include target objects from an image based on the use of a region proposal generation method. The second stage is to extract features of the extracted region proposals using a neural network, and then identify the class of the target object in each region proposal by a classifier.

[0006] In related technologies, when an image of a subject is taken, the camera can be in a landscape mode or rotated to a specific angle in a specific direction, so the orientation of the subject in the taken image can be different from the actual orientation of the subject, i.e., the taken image is also rotated. For example, when a camera takes an image in a specific orientation, the taken image can be rotated as shown in Figure 1 When detecting such a rotated target image, data augmentation is usually performed, i.e., various geometric transformations are performed on the training data of a neural network in advance to enable the neural network to learn the characteristics of the rotated object, and then the neural network generated by training is used to perform target detection. Because data augmentation is needed, this implementation process is relatively complex, thus wasting a large amount of computing time and computing resources. SUMMARY

[0007] Embodiments of the first aspect provide a target detection method. The method includes: obtaining a first scene image taken by a camera; obtaining current position and attitude information of the camera; adjusting the first scene image based on the current position and attitude information of the camera to obtain a second scene image; and performing target detection on the second scene image.

[0008] Embodiments of the second aspect provide a target detection apparatus. The apparatus comprises: a first obtaining module configured to obtain a first scene image captured by a camera; a second obtaining module configured to obtain current position and attitude information of the camera; an adjusting module configured to adjust the first scene image based on the current position and attitude information of the camera to obtain a second scene image; and a detecting module configured to perform target detection on the second scene image.

[0009] Embodiments of the third aspect provide a terminal device, comprising: a memory, a processor and a computer program, the computer program being stored in the memory and executable by the processor. When the processor executes the computer program, the target detection method according to the embodiments of the first aspect is implemented.

[0010] Embodiments of the fourth aspect provide a computer readable storage medium, the computer readable storage medium storing a computer program. When the computer program is executed by a processor, the target detection method according to the embodiments of the first aspect is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and / or additional aspects and advantages of embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, wherein:

[0012] Figure 1 is a schematic diagram of a rotated image according to embodiments of the disclosure;

[0013] Figure 2 is a flowchart of a target detection method according to embodiments of the disclosure;

[0014] Figure 3 is a schematic diagram of an unadjusted scene image according to embodiments of the disclosure;

[0015] Figure 4 is a schematic diagram of an adjusted scene image according to embodiments of the disclosure;

[0016] Figure 5 is a flowchart of a target detection method according to embodiments of the disclosure;

[0017] Figure 6 is a flowchart of a method of generating a plurality of three-dimensional regions according to embodiments of the disclosure;

[0018] Figure 7 is a flowchart of a method of generating a candidate region according to embodiments of the disclosure;

[0019] Figure 8 is a block diagram of a target detection apparatus according to embodiments of the disclosure;

[0020] Figure 9 is a block diagram of a target detection device according to another embodiment of the present disclosure;

[0021] Figure 10 is a block diagram of a terminal device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in detail, and examples of the embodiments are illustrated in the accompanying drawings. Throughout the specification, the same or similar elements and elements having the same or similar functions are denoted by the same reference numerals. The embodiments described herein with reference to the accompanying drawings are explanatory for explaining the present disclosure, and are not to be construed as limiting the embodiments of the present disclosure.

[0023] In detecting a rotated target image, data augmentation is usually performed, that is, various geometric transformations are performed on the training data of a neural network in advance, so that the neural network can learn the characteristics of a rotated object, and then a generated neural network is used to perform target detection. Due to the need for data augmentation, the implementation process is relatively complex, thereby wasting a large amount of computing time and computing resources. For this reason, an embodiment of the present disclosure provides a target detection method. With this method, by obtaining a first scene image photographed by a camera, obtaining current position and attitude information of the camera, and adjusting the first scene image based on the current position and attitude information of the camera to obtain an adjusted second scene image, target detection is performed on the second scene image, so that target detection can be performed on a scene image without the need for data augmentation, the process is simple, the computing time and computing resources of target detection are saved, and the efficiency of target detection is improved.

[0024] Next, the target detection method, the target detection device, the terminal device, and the computer-readable storage medium will be described with reference to the accompanying drawings.

[0025] The following describes a target detection method according to an embodiment of the present disclosure. Figure 2 is a flowchart of a target detection method according to an embodiment of the present disclosure. Figure 2 is a flowchart of a target detection method according to an embodiment of the present disclosure.

[0026] As Figure 2 indicated, the target detection method according to the present disclosure can include the following actions.

[0027] At block 101, a first scene image photographed by a camera is obtained.

[0028] In particular, the target detection method according to the present disclosure can be performed by a target detection apparatus according to the present disclosure. The target detection apparatus can be configured in a terminal device to perform target detection on a scene image of a scene. The terminal device according to embodiments of the present disclosure can be any hardware device capable of data processing, such as a smartphone, a tablet computer, a robot, and a wearable device (such as a head-mounted mobile device).

[0029] It can be understood that a camera can be configured in the terminal device to capture the first scene image, so that the target detection apparatus can obtain the first scene image captured by the camera.

[0030] The scene can be an actual scene or a virtual scene. The first scene image can be static or dynamic, which is not limited herein. In addition, the first scene image captured by the camera can be an image without rotation, in which the object has an orientation consistent with the actual orientation of the object, or the first scene image captured by the camera can be an image with rotation, in which the object has an orientation inconsistent with the actual orientation of the object, which is not limited herein.

[0031] At block 102, current position and pose information of the camera is obtained.

[0032] At block 103, the first scene image is adjusted based on the current position and pose information of the camera to obtain a second scene image.

[0033] The current position and pose information can include an orientation of the camera.

[0034] In the specific implementation, a simultaneous localization and mapping (SLAM, hereinafter referred to as SLAM) system can be used to obtain the current position and pose information of the camera.

[0035] The SLAM system used in embodiments of the present disclosure will be briefly described below.

[0036] The SLAM system, as its name implies, can realize localization and mapping. When a user holds or wears a terminal device from an unknown position in an unknown environment, the SLAM system in the terminal device estimates the position and pose of the camera at each time based on the feature points monitored by the camera during the movement, and fuses the image frames obtained by the camera at different times to reconstruct a complete three-dimensional map of the scene around the user. The SLAM system is widely used in robot positioning and navigation, virtual reality (VR), augmented reality (AR), unmanned aerial vehicles, and self-driving. The position and pose of the camera at each time can be represented by a matrix or a vector containing rotation and translation information.

[0037] A SLAM system can generally be divided into a visual front-end module and an optimization back-end module.

[0038] The main task of the visual front-end module is to solve the camera pose transformation between adjacent frames through feature matching using image frames acquired by the camera at different time instants during motion, and to reconstruct the map by fusing the image frames.

[0039] The visual front-end module relies on terminal devices, such as sensors installed in robots or smartphones. Common sensors include cameras (e.g., monocular cameras, binocular cameras, Time of Flight (TOF) cameras), Inertial Measurement Units (IMUs), and lidars, which are configured to collect various types of raw data in the actual environment, including laser scanning data, video image data, and point cloud data.

[0040] The optimization back-end module of the SLAM system is mainly to optimize and fine-tune the inaccurate camera pose and reconstructed map obtained by the visual front-end module. The optimization back-end module can be separated from the visual front-end module as an offline operation, or integrated into the visual front-end module.

[0041] Current SLAM systems are generally based on visual-inertial odometry (VIO), which tracks the position and orientation of a camera by synchronously processing visual signals and Inertial Measurement Unit (IMU) signals.

[0042] The following is a brief introduction to the process of determining the position and pose information of a camera by a SLAM system.

[0043] Initialization is performed. Specifically, feature points can be identified from a scene image acquired by a camera, the feature points extracted from the scene image and acquired at different times are associated to find a correspondence between the feature points extracted at different times, and a three-dimensional position of the feature points and a positional relationship of the camera can be calculated according to the correspondence.

[0044] After initialization, when the camera acquires content that has not been acquired previously, the SLAM system can track the camera pose in real time and incrementally expand the number of three-dimensional points.

[0045] In addition, after the current position and pose information of the camera is acquired, the first scene image can be adjusted based on the current position and pose information of the camera to obtain an adjusted second scene image.

[0046] Specifically, when the first scene image is adjusted, a rotation angle of the first scene image can be determined based on the current position and pose information of the camera, so that the first scene image is rotated based on the rotation angle to obtain the second scene image. The orientation of the object in the second scene image is the same as the actual orientation of the object. In other words, the horizontal direction in the second scene image is parallel to the transverse direction of the second scene image.

[0047] For example, assume Figure 3 is a first scene image taken by a camera. According to the current position and pose information of the camera acquired by a SLAM system when the camera takes the image, it is determined that the rotation angle of the first scene image is 45 degrees clockwise, and then the first scene image can be rotated 45 degrees counterclockwise, so that Figure 3 the horizontal direction (direction B in Figure 3 ) of the first scene image is parallel to the transverse direction (direction A in Figure 3 ) of the first scene image, to obtain Figure 4 the second scene image shown in

[0048] It should be noted that the technology of acquiring the current position and pose information of the camera by the SLAM system is relatively mature, and will not be repeated here.

[0049] At block 104, target detection is performed on the second scene image.

[0050] Specifically, after the first scene image is adjusted to obtain the second scene image, the second scene image can be divided into a plurality of candidate regions. The feature maps of the plurality of candidate regions are extracted using a neural network. The class of the object in each candidate region is identified using a classification method, and the size of each object is determined by performing bounding box regression on each object, so that target detection on the plurality of candidate regions in the second scene image can be realized to determine the target object to be detected in the second scene image. Since the second scene image is a scene image obtained by adjusting the direction of the first scene image, the target detection result of the second scene image is the target detection result of the first scene image, except that the target object to be detected has a different orientation in the first scene image and the second scene image.

[0051] The neural network used to extract the feature map of the candidate region can be any neural network used to extract features, the class of the object can be determined by using any neural network used for image classification, and when performing bounding box regression, any neural network used for bounding box regression can be used, which is not limited here.

[0052] It is worth noting that in the embodiments of the present disclosure, the direction of the second scene image is related to the training data of the neural network used to perform target detection on the second scene image. For example, in the embodiments of the present disclosure, in the training data used to train the neural network, the transverse direction of the image is usually the horizontal direction in the image, and when the first scene image is adjusted, the first scene image is adjusted so that the horizontal direction of the adjusted image can correspondingly be parallel to the transverse direction of the adjusted image. That is, the orientation of the object in the training data used to train the neural network is the same as the orientation of the object in the second scene image. In specific embodiments, the first scene image can be adjusted to have other directions as needed, which is not limited herein.

[0053] In addition, it can be understood that the orientation of the object in the first scene image taken by the camera can be the same as or different from the orientation of the object in the training data used to train the neural network. In the embodiments of the present disclosure, if it is determined that the object in the first scene image taken by the camera has the same orientation as the object in the training data used to train the neural network, target detection can be directly performed on the first scene image.

[0054] That is, in the embodiments of the present disclosure, before adjusting the first scene image, the method further comprises: determining that the first scene image meets the adjustment requirement based on the current position and attitude information of the camera. The adjustment requirement can be that the rotation angle of the first scene image is greater than 0 degrees.

[0055] Specifically, if it is determined based on the current position and attitude information of the camera that the first scene image taken by the camera meets the adjustment requirement, the first scene image can be adjusted based on the current position and attitude of the camera to obtain an adjusted second scene image. Then, the second scene image is subjected to target detection. If the first scene image does not meet the adjustment requirement, target detection is directly performed on the first scene image taken by the camera.

[0056] In exemplary embodiments, an angle threshold can be set, and the adjustment requirement can be set to be that the rotation angle of the first scene image is greater than the angle threshold, which is not limited herein.

[0057] It can be understood that in the target detection method according to the present disclosure, since the first scene image is adjusted based on the current position and attitude of the camera before target detection is performed, an adjusted second scene image is obtained. The orientation of the object in the second scene image is the same as the orientation of the object in the training data of the neural network, so various conversions of the training data of the neural network are not required in advance to enable the neural network to learn the characteristics of the rotated object. The neural network generated using the training data trained in a single direction can be directly used to perform target detection on the second scene image, thereby simplifying the process, saving the calculation time and calculation resources of target detection, and improving the efficiency of target detection.

[0058] With the target detection method according to the embodiments of the present disclosure, after a first scene image captured by a camera is acquired, current position and attitude information of the camera is acquired, the first scene image is adjusted based on the current position and attitude information of the camera to obtain an adjusted second scene image, and target detection is performed on the second scene image. In this way, target detection can be performed on a scene image without the need for data augmentation, the process is simple, and the computing time and computing resources for target detection are saved, improving the efficiency of target detection.

[0059] According to the above analysis, after the first scene image is adjusted to obtain the adjusted second scene image, the second scene image can be directly divided into a plurality of candidate regions by the method described in the above embodiments, and then subsequent target detection is performed. In possible implementation manners, in order to improve the accuracy of the generated candidate regions, a three-dimensional point cloud corresponding to the second scene image can be acquired by a SLAM system, the second scene image is divided by using the three-dimensional point cloud to form a plurality of candidate regions, and subsequent target detection is performed. The following will be described with reference to Figure 5 The target detection method according to the embodiments of the present disclosure is further described. Figure 5 The flowchart of the target detection method of another embodiment of the present disclosure.

[0060] As Figure 5 shown, the target detection method according to the present disclosure can include the following steps.

[0061] At block 201, a first scene image captured by a camera is acquired.

[0062] At block 202, current position and attitude information of the camera is acquired by a simultaneous localization and mapping (SLAM) system.

[0063] At block 203, a rotation angle of the first scene image is determined based on the current position and attitude information of the camera.

[0064] At block 204, the first scene image is rotated based on the rotation angle to obtain a second scene image.

[0065] The specific implementation process and principles of the actions of blocks 201 to 204 described above can refer to the description of the above embodiments, which will not be described here.

[0066] At block 205, a scene corresponding to the first scene image is scanned by the SLAM system to generate a three-dimensional point cloud corresponding to the scene.

[0067] Any prior art can scan the scene corresponding to the first scene image by the SLAM system to generate a three-dimensional point cloud corresponding to the scene, which is not limited here.

[0068] In an example embodiment, a camera included in a terminal device can be pre-calibrated to determine internal parameters of the camera, and the calibrated camera is used to scan a scene to generate a three-dimensional point cloud corresponding to the scene by a SLAM system.

[0069] To calibrate the camera, a 7*9 black-and-white calibration version can be printed on an A4 paper, and one chessboard size of the calibration board is 29.1mm*29.1mm. The calibration board is pasted on a neat and flat wall, and a video of the calibration board is taken by using the camera to be calibrated. During the shooting, the camera is continuously moved to take the calibration board from different angles and different distances. A calibration program is written using the OpenCV encapsulated algorithm function. The video is converted into images, and 50 images are selected as calibration images. The calibration images are input into the calibration program together with the basic parameters of the calibration board, and the internal parameters of the camera can be calculated.

[0070] Points in the world coordinate system are measured in physical length dimensions, while points in the image plane coordinate system are measured in pixels. The internal parameters are used to linearly transform between the two coordinate systems. A point Q(X, Y, Z) in space can be transformed by the internal parameter matrix to obtain the corresponding point Q(u, v) of the point in the pixel coordinate system projected onto the image plane by a ray:

[0071]

[0072] K is the internal parameter matrix of the camera.

[0073]

[0074] Where f is the focal length of the camera, in millimeters; dx and dy represent the length and width of each pixel, respectively, in millimeters; and u0 and v0 represent the coordinates of the image center, usually in pixels.

[0075] According to the internal parameters of the camera and the height and width of the scene image obtained when the camera shoots the scene, a camera parameter file is written in the format required by the DSO program, and the DSO program is started as input. In other words, when the camera is used to scan the scene, a three-dimensional point cloud of the scene can be constructed in real time.

[0076] It should be noted that the action at block 205 can be performed after the action at block 204, or can be performed before the action at block 204, which is not limited herein, and the action at block 205 only needs to be performed before the action at block 206.

[0077] At block 206, the three-dimensional point cloud is adjusted based on the current position and attitude information of the camera, so that the adjusted three-dimensional point cloud corresponds to the direction of the second scene image.

[0078] Specifically, the direction of the three-dimensional point cloud corresponding to the scene is adjusted in a similar manner as the action at block 103, so that the adjusted three-dimensional point cloud corresponding to the scene corresponds to the direction of the second scene image to be performed target detection.

[0079] It should be noted that in the embodiments of the present disclosure, when generating the three-dimensional point cloud corresponding to the second scene image, the three-dimensional point cloud corresponding to the first scene image is formed, and the three-dimensional point cloud is adjusted based on the current position and attitude information of the camera, so that the three-dimensional point cloud corresponds to the direction of the second scene image. In the exemplary embodiments, after obtaining the current position and attitude information of the camera by the SLAM system, the current position and attitude information of the camera determined by the SLAM system is directly used to scan the scene corresponding to the second scene image to directly generate the three-dimensional point cloud corresponding to the second scene image, which is not limited herein.

[0080] At block 207, the second scene image is divided based on the adjusted three-dimensional point cloud to form a plurality of candidate regions.

[0081] Specifically, the action at block 207 can be achieved by the following steps.

[0082] At block 207a, the adjusted three-dimensional point cloud is divided to form a plurality of three-dimensional regions.

[0083] At block 207b, the plurality of three-dimensional regions are projected to the second scene image to form a plurality of candidate regions.

[0084] Specifically, the action at block 207 can be achieved in the following manner.

[0085] First way

[0086] It can be understood that the same object usually has the same or similar texture, color and other characteristics, but different objects have different texture, color and other characteristics. Correspondingly, in the adjusted three-dimensional point cloud, the similarity between the three-dimensional points corresponding to the same object is usually greater than the similarity between the three-dimensional points of the object and the three-dimensional points of another object. Then, in the embodiments of the present disclosure, when the adjusted three-dimensional point cloud is divided to form a plurality of three-dimensional regions, based on the similarity between the three-dimensional points in the adjusted three-dimensional point cloud, the three-dimensional points with high similarity (the higher the similarity, the closer the three-dimensional points) are merged together, so that a plurality of three-dimensional point sub-clouds can be formed, and the region where each three-dimensional point sub-cloud is located is configured as a three-dimensional region, thereby dividing the three-dimensional point cloud into a plurality of three-dimensional regions.

[0087] Specifically, the three-dimensional points in the three-dimensional point cloud can be classified into multiple categories by using a clustering algorithm, so that the similarity between the three-dimensional points of the same category is greater than the similarity between the three-dimensional points of one category and the three-dimensional points of another category. The three-dimensional points of the same category are merged together, so that multiple three-dimensional point sub-clouds can be formed, and the area occupied by one three-dimensional point sub-cloud is configured as a three-dimensional area, so as to divide the three-dimensional point cloud into multiple three-dimensional areas.

[0088] The clustering algorithm can be a distance-based clustering algorithm, such as a k-means clustering algorithm, or a graph-based clustering algorithm, such as a graph cut algorithm, or any other clustering algorithm, which is not limited in the present disclosure.

[0089] The action in block 207a can be implemented in the following manner.

[0090] The three-dimensional points in the three-dimensional point cloud are merged by the clustering algorithm, and the merged three-dimensional point cloud is divided to form multiple three-dimensional areas.

[0091] For example, it is assumed that Figure 6 The three-dimensional points shown in FIG. 6 are part of the three-dimensional points in the adjusted three-dimensional point cloud. In Figure 6 In FIG. 7, the three-dimensional points in the three-dimensional frame 1 are classified into one category, the three-dimensional points in the three-dimensional frame 2 are classified into one category, the three-dimensional points in the three-dimensional frame 3 are classified into one category, and the three-dimensional points in the three-dimensional frame 4 are classified into one category by using the clustering algorithm. The three-dimensional points in the three-dimensional frames 1, 2, 3, and 4 can be merged to form four three-dimensional point sub-clouds, and the area occupied by each three-dimensional point sub-cloud is configured as a three-dimensional area, so as to divide the merged three-dimensional point cloud into four three-dimensional areas.

[0092] The three-dimensional points in the three-dimensional point cloud are merged by the clustering algorithm, and as described below, by taking the K-means algorithm as the clustering algorithm, the merged three-dimensional point cloud is divided to form multiple three-dimensional areas.

[0093] Specifically, the number of three-dimensional regions to be formed can be preset. The three-dimensional points in the three-dimensional point cloud are classified into k categories by the k-means algorithm, and the number N of the three-dimensional points in the three-dimensional point cloud is counted. k three-dimensional cluster center points are randomly generated, and it is determined which of the k three-dimensional cluster center points each of the N three-dimensional points corresponds to, i.e., the category of each three-dimensional point is determined and the three-dimensional points belonging to the category of each cluster center point are determined. For each cluster center point, the coordinates of the center point of all the three-dimensional points belonging to the category of the cluster center point are determined, and the coordinates of the cluster center point are modified to the coordinates of the center point. It is again determined which of the k cluster center points each three-dimensional point corresponds to, and the coordinates of each cluster center point are determined according to the coordinates of the center point of all the three-dimensional points belonging to the category of the cluster center point. The above process is repeated until the algorithm converges. In this way, all the three-dimensional points can be divided into k categories, and the three-dimensional points in each category can be merged together, so that k three-dimensional point sub-clouds can be formed, and the area occupied by each three-dimensional point sub-cloud can be configured as a three-dimensional region, thereby achieving the division of the merged three-dimensional point cloud into k three-dimensional regions.

[0094] When determining which of the k three-dimensional cluster center points a particular three-dimensional point corresponds to, the distance between the three-dimensional point and each of the k cluster center points can be calculated, and the cluster center point with the shortest distance to the three-dimensional point is regarded as the cluster center point corresponding to the three-dimensional point.

[0095] The second way

[0096] It can be understood that an object usually has a specific shape, for example, a cup can be cylindrical, and a door can be square. Correspondingly, for an object in a scene with a specific shape, the three-dimensional points in the corresponding three-dimensional point cloud can also be fitted to a specific shape. In an embodiment of the present disclosure, the three-dimensional points in the three-dimensional point cloud described above can be fitted to a plurality of preset models to divide the three-dimensional point cloud into a plurality of three-dimensional regions, which correspond to the plurality of preset models respectively.

[0097] The action at block 207a can be implemented in the following manner.

[0098] The plurality of three-dimensional points in the three-dimensional point cloud described above can be fitted to a plurality of preset models to divide the three-dimensional point cloud into a plurality of three-dimensional regions corresponding to the plurality of preset models respectively.

[0099] The preset model can be a preset geometric basic model, such as a sphere, a cylinder, or a plane, or can be a complex geometric model composed of geometric basic models, or can be any other preset model, which is not limited here.

[0100] In the specific embodiment, if the three-dimensional points in the three-dimensional point cloud can fit multiple preset models, the three-dimensional points corresponding to the multiple preset models can be merged into multiple three-dimensional point sub-clouds, the three-dimensional points in one three-dimensional point sub-cloud correspond to one preset model, and the area occupied by each three-dimensional point sub-cloud is configured as a three-dimensional area, so that the three-dimensional point cloud can be divided into multiple three-dimensional areas corresponding to the multiple preset models respectively.

[0101] The manner of fitting the three-dimensional points in the three-dimensional point cloud with the preset models can be least square method or any other manner, which is not limited herein.

[0102] For example, assuming that in the adjusted three-dimensional point cloud, the three-dimensional points identified as 1-200 are part of the three-dimensional points, the three-dimensional points identified as 1-100 can be fitted with preset model 1, and the three-dimensional points identified as 101-200 can be fitted with preset model 2, the three-dimensional points identified as 1-100 can be merged into three-dimensional sub-point cloud A, and the three-dimensional points identified as 101-200 can be merged into three-dimensional sub-point cloud B. The area occupied by the three-dimensional sub-point cloud A is configured as a three-dimensional area, and the area occupied by the three-dimensional sub-point cloud B is also configured as a three-dimensional area.

[0103] Taking a cylinder as one of the preset models as an example, when fitting the three-dimensional points in the three-dimensional point cloud with the cylinder, the cylinder is parameterized, for example, the cylinder in space can be represented by parameters such as center coordinates (X, Y, Z) in three-dimensional space, bottom radius, height, orientation, etc. A number of three-dimensional points are randomly selected from the three-dimensional point cloud using a RANdom SAmple Consensus (RANSAC) algorithm. Assuming that these three-dimensional points are on a cylinder, the parameters of the cylinder are calculated, the number of three-dimensional points on the cylinder in all three-dimensional points in the three-dimensional point cloud is counted, and it is determined whether the number exceeds a preset number threshold. If not, a number of three-dimensional points are selected again to repeat the above process; otherwise, it can be determined whether the three-dimensional points on the cylinder and the three-dimensional points in the three-dimensional point cloud can fit the cylinder, and the algorithm continues to determine whether the three-dimensional points in the three-dimensional point cloud can fit other preset models, so that the three-dimensional points fitted with multiple preset models respectively are merged to form multiple three-dimensional point sub-clouds. The three-dimensional points in each three-dimensional point sub-cloud correspond to one preset model, and the area occupied by each three-dimensional point sub-cloud is configured as a three-dimensional area, so that the three-dimensional point cloud can be divided into multiple three-dimensional areas corresponding to the multiple preset models respectively.

[0104] The number threshold can be set as needed, which is not limited herein.

[0105] In addition, a distance threshold can be set. The distance of each three-dimensional point in the three-dimensional point cloud to the cylinder can be calculated, and the three-dimensional points with a distance less than the distance threshold are determined as the three-dimensional points on the cylinder.

[0106] It should be noted that the above first and second manners are only two examples of dividing the adjusted three-dimensional point cloud to form multiple three-dimensional regions. In actual applications, those skilled in the art can divide the adjusted three-dimensional point cloud in any other manner, and the present disclosure is not limited thereto.

[0107] Further, after dividing the adjusted three-dimensional point cloud to form multiple three-dimensional regions, the multiple three-dimensional regions are projected onto the second scene image, and the obtained two-dimensional bounding boxes corresponding to the three-dimensional regions respectively are configured to indicate multiple candidate regions to be determined in the present disclosure.

[0108] Specifically, coordinate transformation can be adopted to convert the coordinates of each three-dimensional point in the three-dimensional region from the object coordinate system to the world coordinate system, the camera coordinate system, the projection coordinate system, and the image coordinate system in sequence. In this way, each three-dimensional region is projected onto the scene image. After the projection, the two-dimensional bounding box corresponding to the three-dimensional region is configured to indicate the candidate region, thereby generating multiple candidate regions.

[0109] For example, it is assumed that the cube in Figure 7 is a three-dimensional region formed by dividing the adjusted three-dimensional point cloud. After projecting the three-dimensional region onto the second scene image, the two-dimensional bounding box corresponding to the three-dimensional region (indicated by the dashed box 5 in Figure 7 ) is configured to indicate the candidate region.

[0110] It can be understood that when performing target detection in the embodiments of the present disclosure, the three-dimensional point cloud corresponding to the scene generated by scanning the scene using the SLAM system is divided to form multiple candidate regions. By combining three-dimensional information, the generated candidate regions can be more accurate and less in number.

[0111] It should be noted that in the foregoing embodiments, after generating the three-dimensional point cloud corresponding to the scene by scanning the scene using the SLAM system, the adjusted three-dimensional point cloud is divided to form multiple three-dimensional regions, and the multiple three-dimensional regions are projected onto the adjusted second scene image to form multiple candidate regions. In actual applications, the dense three-dimensional point cloud corresponding to the scene can be obtained by a depth camera, or the three-dimensional point cloud of the scene can be obtained by other methods, and the three-dimensional point cloud can be adjusted, divided, and subsequently operated to form multiple candidate regions, and the present disclosure is not limited thereto.

[0112] At block 208, target detection is respectively performed on the multiple candidate regions.

[0113] Specifically, after the plurality of candidate regions are formed, a feature map of the plurality of candidate regions can be extracted by using a neural network, and a classification method is adopted to identify the category of the object in each candidate region. A bounding box regression is performed on each object to determine the size of each object, so as to realize target detection on the plurality of candidate regions, and determine the target object to be detected in the scene image.

[0114] The neural network used to extract the feature map of the candidate region can be any neural network used to extract features, any neural network used to classify images can be used to determine the category of the object, and when performing the bounding box regression, any neural network used for the bounding box regression can be used, which is not limited herein.

[0115] It can be understood that the target detection method according to the embodiments of the present disclosure can be applied in an AR software development kit (SDK) to provide a target detection function, and a developer can utilize the target detection function in the AR SDK to realize the identification of objects in a scene, and further realize various functions such as product recommendation in the field of e-commerce.

[0116] With the target detection method of the embodiments of the present disclosure, before performing target detection on the first scene image, the first scene image is adjusted based on the current position and attitude information of the camera to obtain an adjusted second scene image, and target detection is performed on the second scene image, so that target detection can be performed on the scene image without data augmentation, the process is simple, the computing time and computing resources of target detection are saved, and the efficiency of target detection is improved. In addition, the three-dimensional point cloud corresponding to the scene generated by the SLAM system is used to assist in generating a plurality of candidate regions, so that the generated candidate regions are more accurate and less in number. Since the number of candidate regions is reduced, the computing time spent on subsequent processing such as feature extraction of the candidate regions is less, and the computing resources consumed are less, thereby saving the computing time and computing resources of target detection, and improving the efficiency of target detection.

[0117] Reference is made below to Figure 8 a target detection device according to an embodiment of the present disclosure. Figure 8 is a block diagram of a target detection device according to an embodiment of the present disclosure.

[0118] As Figure 8 shown, the target detection device includes a first acquisition module 11, a second acquisition module 12, an adjustment module 13, and a detection module 14.

[0119] The first acquisition module 11 is configured to acquire a first scene image photographed by a camera.

[0120] The second acquisition module 12 is configured to acquire current position and pose information of the camera.

[0121] The adjustment module 13 is configured to adjust the first scene image based on the current position and pose information of the camera to obtain a second scene image.

[0122] The detection module 14 is configured to perform target detection on the second scene image.

[0123] In detail, the target detection apparatus can perform the target detection method described in the foregoing embodiments. The apparatus can be configured to perform target detection on a scene image of a scene in a terminal device. The terminal device in the embodiments of the present disclosure can be any hardware device capable of data processing, such as a smartphone, a tablet computer, a robot, a wearable device such as a head-mounted mobile device.

[0124] In an exemplary embodiment, the second acquisition module 12 is configured to acquire the current position and pose information of the camera by a SLAM system.

[0125] In an exemplary embodiment, the adjustment module 13 is configured to determine a rotation angle of the first scene image based on the current position and pose information of the camera; and rotate the first scene image based on the rotation angle.

[0126] It should be noted that the implementation process and technical principles of the target detection apparatus of the present embodiment are described with reference to the description of the target detection method in the embodiments of the first aspect above, and will not be described herein again.

[0127] The target detection apparatus according to the embodiments of the present disclosure acquires a first scene image captured by a camera, acquires current position and pose information of the camera, and then adjusts the first scene image based on the current position and pose information of the camera to obtain an adjusted second scene image, and performs target detection on the second scene image. In this way, target detection can be performed on a scene image without the need for data augmentation, the process is simple, the computing time and computing resources for target detection are saved, and the efficiency of target detection is improved.

[0128] The target detection apparatus according to the embodiments of the present disclosure will be further described below. Figure 9 Further description of the target detection apparatus according to the embodiments of the present disclosure. Figure 9 is a block diagram of a target detection apparatus according to another embodiment of the present disclosure.

[0129] As Figure 9 shown, in Figure 8On the basis of the foregoing, the device further includes a processing module 15 configured to scan a scene corresponding to the first scene image by a SLAM system to generate a three-dimensional point cloud corresponding to the scene, and adjust the three-dimensional point cloud according to the current position and attitude information of the camera, so that the adjusted three-dimensional point cloud corresponds to the direction of the second scene image.

[0130] The detection module 14 includes a division unit 141 and a detection unit 142.

[0131] The division unit 141 is configured to divide the second scene image to form a plurality of candidate regions.

[0132] The detection unit 142 is configured to respectively perform target detection on the plurality of candidate regions.

[0133] In an exemplary embodiment, the division unit 141 is configured to divide the second scene image based on the adjusted three-dimensional point cloud to form a plurality of candidate regions.

[0134] In an exemplary embodiment, the division unit 141 is configured to divide the adjusted three-dimensional point cloud to form a plurality of three-dimensional regions, and project the plurality of three-dimensional regions to the second scene image to form a plurality of candidate regions.

[0135] It should be noted that the implementation process and technical principles of the target detection device of the present embodiment are described above with reference to the description of the target detection method of the first aspect of the embodiment, and will not be described here.

[0136] According to the target detection method of the embodiment of the present disclosure, before performing target detection on the first scene image, the first scene image is adjusted based on the current position and attitude information of the camera to obtain an adjusted second scene image, and target detection is performed on the second scene image, so that target detection can be performed on the scene image without the need for data enhancement, the process is simple, and the computing time and computing resources of target detection are saved, and the efficiency of target detection is improved. In addition, the three-dimensional point cloud corresponding to the scene generated by the SLAM system is used to assist in generating a plurality of candidate regions, so that the generated candidate regions are more accurate and less in number. Since the number of candidate regions is reduced, the computing time spent on subsequent processing such as feature extraction of the candidate regions is less, and the computing resources consumed are less, thereby saving the computing time and computing resources of target detection, and improving the efficiency of target detection.

[0137] In order to implement the above-mentioned embodiments, the present disclosure further provides a terminal device.

[0138] Figure 10 is a block diagram of a terminal device according to an embodiment of the present disclosure.

[0139] As Figure 10As shown, the terminal device comprises a memory, a processor and a computer program, the computer program being stored in the memory and executable by the processor. When the processor executes the computer program, the target detection method according to the above Figure 2 The target detection method of the embodiments described above.

[0140] It should be noted that the implementation process and technical principles of the terminal device of the embodiments are described with reference to the description of the target detection method of the embodiments described above, and will not be repeated here. Figure 2

[0141] The terminal device according to the embodiments of the present disclosure, after obtaining a first scene image photographed by a camera, obtains current position and attitude information of the camera, and adjusts the first scene image based on the current position and attitude information of the camera to obtain an adjusted second scene image, and performs target detection on the second scene image. In this way, target detection can be performed on a scene image without the need for data enhancement, the process is simple, and the computing time and computing resources of target detection are saved, and the efficiency of target detection is improved.

[0142] In order to implement the above-mentioned embodiments, the present disclosure further provides a computer readable storage medium, the computer readable storage medium storing a computer program. When the computer program is executed by a processor, the target detection method according to the embodiments of the first aspect is implemented.

[0143] In order to implement the above-mentioned embodiments, the present disclosure further provides a computer program. When the instructions in the computer program are executed by a processor, the target detection method according to the above-mentioned embodiments is implemented.

[0144] Throughout the specification, references to "embodiment", "some embodiments", "example", "specific example" or "some examples" mean that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure.

[0145] In addition, terms such as "first" and "second" are used herein for the purpose of description and are not intended to indicate or imply relative importance or significance. Therefore, the features defined with "first" and "second" can include one or more of the features.

[0146] Any process or method described in a flowchart or otherwise described herein can be understood as including one or more modules, code segments or code portions for implementing specific logic functions or steps in the process. Those skilled in the art will understand that the scope of the preferred embodiments of the present disclosure includes other implementations.

[0147] ​It is understood that various portions of the disclosure can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, if implemented by hardware, these steps or methods can be implemented by one or a combination of the following technologies known in the art: discrete logic circuitry having logic gates for implementing data signal logic functions, application specific integrated circuits (ASICs) having appropriate combinations of logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0148] As can be understood by those skilled in the art, all or part of the steps performed by the methods in the above-described embodiments can be completed by program instruction-related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, one step or a combination of several steps of the methods in the above-described embodiments can be completed.

[0149] The above-described storage medium can be a read-only memory, a magnetic disk, a CD, or the like. Although illustrative embodiments have been shown and described, it will be understood by those skilled in the art that the above-described embodiments cannot be construed as limiting the disclosure, and changes, replacements, and modifications can be made to the embodiments without departing from the scope of the disclosure.

Claims

1. A target detection method, comprising: obtaining a first scene image captured by a camera, the first scene image comprising an object; obtaining current position and pose information of the camera; adjusting the first scene image based on the current position and pose information of the camera to obtain a second scene image, wherein adjusting the first scene image based on the current position and pose information of the camera comprises determining a rotation angle and a rotation direction of the first scene image based on the current position and pose information of the camera, and rotating the first scene image based on the rotation angle and the rotation direction, an orientation of the object in the second scene image is the same as an actual orientation of the object; and performing target detection on the second scene image, an orientation of the object in the second scene image is the same as an orientation of the object in training data of a neural network.

2. The object detection method of claim 1, wherein, obtaining the current position and pose information of the camera comprises: obtaining the current position and pose information of the camera by a simultaneous localization and mapping (SLAM) system.

3. The target detection method of claim 1, further comprising: determining, based on the current position and pose information of the camera, that the first scene image meets an adjustment requirement, wherein the adjustment requirement refers to that the rotation angle of the first scene image is greater than 0 degrees.

4. The object detection method of claim 2, wherein, performing the target detection on the second scene image comprises: dividing the second scene image to form a plurality of candidate regions; and respectively performing the target detection on the plurality of candidate regions.

5. The target detection method of claim 4, further comprising: scanning, by the SLAM system, a scene corresponding to the first scene image to generate a three-dimensional point cloud corresponding to the scene; and adjusting the three-dimensional point cloud based on the current position and pose information of the camera to make the three-dimensional point cloud correspond to a direction of the second scene image; or scanning, by the SLAM system, a scene corresponding to the second scene image to generate a three-dimensional point cloud corresponding to the scene. scanning the scene comprises:

6. The object detection method of claim 5, wherein, calibrating the camera to determine internal parameters of the camera; and scanning the scene using the calibrated camera to generate, by the SLAM system, the three-dimensional point cloud corresponding to the scene. dividing the second scene image to form the plurality of candidate regions comprises:

7. The object detection method of claim 5, wherein, dividing the second scene image based on the three-dimensional point cloud to form the plurality of candidate regions. dividing the second scene image based on the three-dimensional point cloud to form the plurality of candidate regions comprises:

8. The object detection method of claim 7, wherein, dividing the three-dimensional point cloud to form a plurality of three-dimensional regions; and projecting the plurality of three-dimensional regions to the second scene image to form the plurality of candidate regions. dividing the three-dimensional point cloud to form the plurality of three-dimensional regions comprises:

9. The object detection method of claim 8, wherein, merging three-dimensional points in the adjusted three-dimensional point cloud by a clustering algorithm to obtain a merged three-dimensional point cloud; and dividing the merged three-dimensional point cloud to form the plurality of three-dimensional regions. dividing the three-dimensional point cloud to form the plurality of three-dimensional regions comprises:

10. The object detection method of claim 8, wherein, ​ fitting the three-dimensional points in the adjusted three-dimensional point cloud to a plurality of preset models to divide the three-dimensional point cloud into a plurality of three-dimensional regions, the plurality of three-dimensional regions respectively corresponding to the plurality of preset models.

11. The object detection method of claim 4, wherein, respectively performing the target detection on the plurality of candidate regions comprises: using a classification algorithm to identify a class of each object in a candidate region; and determining a size of the object by performing a bounding box regression on the object to achieve the target detection on the candidate region.

12. A terminal device comprising: a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein when the processor executes the computer program, the processor implements a target detection method, the target detection method comprising: obtaining a first scene image captured by a camera, the first scene image including an object; obtaining current position and attitude information of the camera; adjusting the first scene image based on the current position and attitude information of the camera to obtain a second scene image, wherein adjusting the first scene image based on the current position and attitude information of the camera comprises determining a rotation angle and a rotation direction of the first scene image based on the current position and attitude information of the camera, and rotating the first scene image based on the rotation angle and the rotation direction; an orientation of the object in the second scene image is the same as an actual orientation of the object; and performing target detection on the second scene image.

13. The terminal device of claim 12, wherein, obtaining the current position and attitude information of the camera comprises: obtaining the current position and attitude information of the camera by a simultaneous localization and mapping (SLAM) system.

14. The terminal device of claim 12, wherein, adjusting the first scene image based on the current position and attitude information of the camera comprises: determining a rotation angle of the first scene image based on the current position and attitude information of the camera; and rotating the first scene image based on the rotation angle.

15. The terminal device of claim 13, wherein performing the target detection on the second scene image comprises: dividing the second scene image to form a plurality of candidate regions; and respectively performing target detection on the plurality of candidate regions.

16. The terminal device of claim 15, wherein the target detection method further comprises: scanning a scene corresponding to the first scene image by the SLAM system to generate a three-dimensional point cloud corresponding to the scene; and adjusting the three-dimensional point cloud based on the current position and attitude information of the camera to make the three-dimensional point cloud correspond to a direction of the second scene image; or scanning a scene corresponding to the second scene image by the SLAM system to generate a three-dimensional point cloud corresponding to the scene. dividing the second scene image to form the plurality of candidate regions comprises:

17. The terminal device of claim 16, wherein, dividing the second scene image based on the three-dimensional point cloud to form the plurality of candidate regions. dividing the second scene image based on the three-dimensional point cloud to form the plurality of candidate regions comprises:

18. The terminal device of claim 17, wherein, dividing the three-dimensional point cloud to form a plurality of three-dimensional regions; and ​ projecting the plurality of three-dimensional regions to the second scene image to form the plurality of candidate regions.

19. A non-transitory computer-readable storage medium having stored therein a computer program, wherein, When the computer program is executed by a processor, the processor implements a target detection method, the target detection method comprising: obtaining a first scene image photographed by a camera, the first scene image comprising an object; obtaining current position and attitude information of the camera; adjusting the first scene image based on the current position and attitude information of the camera to obtain a second scene image, wherein adjusting the first scene image based on the current position and attitude information of the camera comprises determining a rotation angle and a rotation direction of the first scene image based on the current position and attitude information of the camera, and rotating the first scene image based on the rotation angle and the rotation direction; an orientation of the object in the second scene image is the same as an actual orientation of the object; and performing target detection on the second scene image.

Images