Target object detection method, device, equipment and storage medium

Abstract

The present disclosure provides a target object detection method and device, equipment and a storage medium, relating to the technical field of artificial intelligence and computer vision. The method comprises: performing feature extraction on a to-be-detected image, object depth information and object normal vector information respectively to obtain image features, depth features and normal vector features, wherein the to-be-detected image, object depth information and object normal vector information are all associated with at least one target object; fusing the image features, depth features and normal vector features to obtain initial fusion features; extracting feature information of the initial fusion features in at least one preset direction to obtain at least one direction fusion feature; and performing target object detection on the at least one target object according to the at least one direction fusion feature to obtain a target object detection result.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence technology and computer vision technology, and in particular to a method, apparatus, device, medium and program product for target object detection. Background Technology

[0002] With the rapid development of artificial intelligence technology and the widespread adoption of intelligent robots, business establishments such as sales halls often require mobile intelligent service robots to assist sales staff in providing services to customers. In sales halls, where there are many customers, intelligent service robots typically rely on monocular image acquisition devices to identify the location of customers and other objects, aiming to accurately avoid them. However, the inventors discovered that in situations with high foot traffic, intelligent service robots struggle to achieve accurate obstacle avoidance, sometimes resulting in collisions with customers or objects such as luggage. Summary of the Invention

[0003] In view of the above problems, this disclosure provides target object detection methods, apparatus, equipment, media and program products.

[0004] According to a first aspect of this disclosure, a target object detection method is provided, comprising:

[0005] Feature extraction is performed on the image to be detected, the object depth information, and the object normal vector information to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object.

[0006] By fusing the above image features, the above depth features, and the above normal vector features, an initial fused feature is obtained;

[0007] Extract the feature information of the above initial fusion features in at least one preset direction to obtain at least one directional fusion feature; and

[0008] Based on at least one of the aforementioned directional fusion features, target object detection is performed on at least one of the aforementioned target objects to obtain target object detection results.

[0009] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the image horizontal direction of the image to be detected, and the direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction.

[0010] The extraction of feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes:

[0011] The above-mentioned initial fusion features are input into the first fusion feature extraction layer, and the above-mentioned first-direction fusion features are output.

[0012] The first fusion feature extraction layer is constructed based on the first convolutional kernel corresponding to the preset horizontal direction.

[0013] According to embodiments of this disclosure, the preset direction includes a preset vertical direction corresponding to the vertical direction of the image to be detected, and the direction fusion feature includes a second direction fusion feature corresponding to the preset vertical direction;

[0014] The extraction of feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes:

[0015] The initial fusion features are input into the second fusion feature extraction layer, and the second directional fusion features are output.

[0016] The second fusion feature extraction layer is constructed based on a second convolutional kernel corresponding to the preset vertical direction.

[0017] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the image horizontal direction of the image to be detected, and a preset vertical direction corresponding to the image vertical direction of the image to be detected. The direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction and a second direction fusion feature corresponding to the preset vertical direction.

[0018] The extraction of feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes:

[0019] The initial fusion features are input into the first fusion feature extraction layer and the second fusion feature extraction layer respectively, and the first directional fusion feature and the second directional fusion feature are output.

[0020] The first fusion feature extraction layer is constructed based on the first convolution kernel corresponding to the preset horizontal direction, and the second fusion feature extraction layer is constructed based on the second convolution kernel corresponding to the preset vertical direction.

[0021] According to embodiments of this disclosure, the above-mentioned target object detection based on at least one of the above-mentioned directional fusion features, to obtain target object detection results, includes:

[0022] Based on the attention mechanism, the first-direction fusion features and the second-direction fusion features are fused to obtain the target fusion features; and

[0023] The above target fusion features are input into the target object detection layer, and the above target object detection results are output.

[0024] According to embodiments of this disclosure, the object depth information includes an object depth image, and the object normal vector information includes an object normal vector image.

[0025] The above-mentioned target object detection methods also include:

[0026] Depth information detection is performed on the above image to be detected to obtain the depth image of the above object;

[0027] Normal vector information is detected on the depth image of the above object to obtain the normal vector image of the above object.

[0028] According to embodiments of this disclosure, the target object detection method further includes:

[0029] Acquire point cloud data corresponding to the above-mentioned image to be detected, wherein the point cloud data to be detected is associated with at least one of the above-mentioned target objects;

[0030] Depth information detection was performed on the aforementioned point cloud data to obtain the depth information of the aforementioned objects; and

[0031] Normal vector information is detected on the depth information of the above objects to obtain the normal vector information of the above objects.

[0032] A second aspect of this disclosure provides a target object detection apparatus, comprising:

[0033] The first feature extraction module is used to extract features from the image to be detected, the object depth information, and the object normal vector information respectively, to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object.

[0034] The fusion module is used to fuse the above image features, the above depth features, and the above normal vector features to obtain the initial fused features;

[0035] The second feature extraction module is used to extract feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature; and

[0036] The target object detection module is used to perform target object detection on at least one target object based on at least one of the above-mentioned directional fusion features, and obtain the target object detection result.

[0037] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0038] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0039] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0040] According to the target object detection method, apparatus, device, medium, and program products provided in this disclosure, by fusing image features, depth features, and normal vector features related to the target object, the obtained initial fused features can contain at least the normal vector information of the target object surface, thereby improving the semantic information richness of the initial fused features. Then, feature information is extracted from the obtained initial fused features in at least one preset direction, so that the obtained directional fused features can more accurately represent the semantic information of the target object in the detection space in the direction corresponding to the preset direction. Furthermore, the target object is detected based on the directional fused features, so that the target object detection result can more accurately represent the position of the target object in the detection space, thereby achieving the technical effect of improving the target object detection accuracy. Attached Figure Description

[0041] The foregoing contents, other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which the accompanying drawings are shown.

[0042] Figure 1 The illustration shows an application scenario of the target object detection method and apparatus according to embodiments of the present disclosure.

[0043] Figure 2 A flowchart illustrating a target object detection method according to an embodiment of the present disclosure is shown schematically.

[0044] Figure 3 A flowchart illustrating a target object detection method according to another embodiment of the present disclosure is shown.

[0045] Figure 4A The schematic diagram illustrates the principle of extracting feature information of initial fusion features in at least one preset direction according to an embodiment of the present disclosure.

[0046] Figure 4B The schematic diagram illustrates the principle of extracting feature information of initial fusion features in at least one preset direction according to another embodiment of the present disclosure.

[0047] Figure 5 A schematic diagram of a target object detection model according to an embodiment of the present disclosure is shown.

[0048] Figure 6A A schematic diagram of an image to be detected according to an embodiment of the present disclosure is shown.

[0049] Figure 6B A schematic diagram of an object depth image according to an embodiment of the present disclosure is shown.

[0050] Figure 6C A schematic diagram of an object normal vector image according to an embodiment of the present disclosure is shown.

[0051] Figure 6D A schematic diagram illustrating the target object detection results according to an embodiment of the present disclosure is shown.

[0052] Figure 7 A schematic block diagram of a target object detection apparatus according to an embodiment of the present disclosure is shown.

[0053] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a target object detection method according to an embodiment of the present disclosure. Detailed Implementation

[0054] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0055] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0056] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0057] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0058] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0059] With the rapid development of artificial intelligence technology and the widespread adoption of intelligent robots, business establishments such as sales halls typically require the deployment of mobile intelligent service robots to assist sales staff in providing services to customers. In sales halls, where there are numerous customers, intelligent service robots usually rely on monocular image acquisition devices to capture images and identify the location of customers and other targets, enabling the robots to accurately avoid them.

[0060] However, the inventors discovered that monocular image-based target detection methods in related technologies struggle to accurately determine the location of target objects, particularly predicting their depth in space, resulting in low detection accuracy. Common improvements involve adding detection devices such as LiDAR to assist in target object detection, but this increases the manufacturing cost of intelligent service robots, and the depth information detection performance still falls short of practical needs.

[0061] Embodiments of this disclosure provide a target object detection method, apparatus, device, medium, and program product. The target object detection method includes: extracting features from an image to be detected, object depth information, and object normal vector information to obtain image features, depth features, and normal vector features, wherein the image to be detected, object depth information, and object normal vector information are all associated with at least one target object; fusing the image features, depth features, and normal vector features to obtain initial fused features; extracting feature information of the initial fused features in at least one preset direction to obtain at least one direction fused features; and performing target object detection on at least one target object based on the at least one direction fused features to obtain a target object detection result.

[0062] According to embodiments of this disclosure, by fusing image features, depth features, and normal vector features related to the target object, the resulting initial fused features can contain at least the normal vector information of the target object's surface, thereby enhancing the semantic information richness of the initial fused features. Then, feature information is extracted from the resulting initial fused features in at least one preset direction, which can make the resulting directional fused features more accurately represent the semantic information of the target object in the detection space corresponding to the preset direction. Furthermore, target object detection is performed on the target object based on the directional fused features, which can make the target object detection results more accurately represent the position of the target object in the detection space, thereby achieving the technical effect of improving the target object detection accuracy.

[0063] Figure 1 The illustration shows an application scenario of the target object detection method and apparatus according to embodiments of the present disclosure.

[0064] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a vehicle 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the vehicle 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0065] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and vehicle 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and vehicle 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0066] The first terminal device 101 and the second terminal device 102 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0067] Vehicle 103 may be equipped with image acquisition and image processing devices. For example, passenger cars, engineering vehicles, or unmanned vehicles may be equipped with image acquisition devices and image processors.

[0068] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the vehicle 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0069] It should be noted that the target object detection method provided in this embodiment can generally be executed by server 105. Correspondingly, the target object detection device provided in this embodiment can generally be located in server 105. The target object detection method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the vehicle 103, and / or server 105. Correspondingly, the target object detection device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the vehicle 103, and / or server 105.

[0070] Alternatively, the target object detection method provided in this embodiment can generally be executed by any one of the first terminal device 101, the second terminal device 102, and the vehicle 103. Correspondingly, the target object detection device provided in this embodiment can generally be disposed in any one of the first terminal device 101, the second terminal device 102, and the vehicle 103.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, vehicles, networks, and servers can be included.

[0072] The following will be based on Figure 1 The described scene, through Figures 2 to 6D The target object detection method of the present disclosure embodiments will be described in detail.

[0073] Figure 2 A flowchart illustrating a target object detection method according to an embodiment of the present disclosure is shown schematically.

[0074] like Figure 2 As shown, the target object detection method in this embodiment includes operations S210 to S240.

[0075] In operation S210, feature extraction is performed on the image to be detected, the object depth information, and the object normal vector information to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object.

[0076] According to embodiments of this disclosure, the image to be detected can be any type of image, such as a picture or video frame, that records the target object in the detection space. The object depth information can be an image, vector, or other information representing the depth information of pixels in the image to be detected. The object normal vector information can be an image, vector, or other information representing the surface normal vector of the target object.

[0077] In one embodiment of this disclosure, the image to be detected may be a monocular image.

[0078] It should be noted that the image to be detected can be obtained by acquiring an image of a target object in the detection space, and the object depth information and object normal vector information can be obtained by performing depth information detection and normal vector information detection on the image to be detected. However, it is not limited to this; the object depth information and object normal vector information can also be obtained through other methods, such as by using a detection device such as LiDAR to detect the target object in the detection space and obtaining the corresponding object depth information and object normal vector information. The embodiments of this disclosure do not limit the specific method of obtaining the object depth information and object normal vector information, and those skilled in the art can choose according to actual needs.

[0079] It should be understood that the number of target objects can be one or more. When there are multiple target objects, the image to be detected, the object depth information, and the object normal vector information can all represent at least one identical target object, so as to achieve target object detection for that identical target object.

[0080] According to embodiments of this disclosure, feature extraction can be performed based on neural network algorithms. For example, convolutional neural networks can be used to extract features from the image to be detected, object depth information, and object normal vector information separately. However, this is not limited to this; other types of neural network algorithms, such as recurrent neural network algorithms, can also be used to extract features from the image to be detected, object depth information, and object normal vector information separately. Embodiments of this disclosure do not limit this approach.

[0081] In operation S220, image features, depth features, and normal vector features are fused to obtain the initial fused features.

[0082] According to embodiments of this disclosure, image features, depth features, and normal vector features can be fused based on deep learning algorithms. For example, image features, depth features, and normal vector features can be fused based on deep learning algorithms such as attention networks and multilayer perceptrons. However, this is not limited to this; other types of algorithms can also be used to fuse image features, depth features, and normal vector features, and embodiments of this disclosure do not limit this approach.

[0083] According to embodiments of this disclosure, the fused features may include at least semantic information that can characterize the depth information of the target object and the normal vector information of the surface of the target object, thereby improving the semantic information richness of the initial fused features and at least partially avoiding the technical problem of losing the semantic information of the target object in the detection space.

[0084] In operation S230, feature information of the initial fusion features in at least one preset direction is extracted to obtain at least one directional fusion feature.

[0085] According to embodiments of this disclosure, feature extraction can be performed based on neural network algorithms, such as convolutional neural networks, to extract features from the initial fused features. Embodiments of this disclosure do not limit this to the specific implementation.

[0086] It should be noted that the embodiments of this disclosure do not limit the specific type of the preset direction. For example, it can be a horizontal direction or a vertical direction corresponding to the image to be detected. However, it is not limited to this, and it can also be other types of directions. For example, it can also be a direction with a preset angle to the horizontal direction corresponding to the image to be detected. The embodiments of this disclosure do not limit this.

[0087] It should be understood that the embodiments of this disclosure do not limit the number of directional fusion features, as long as the requirements are met.

[0088] In operation S240, target object detection is performed on at least one target object based on fusion features in at least one direction, and the target object detection result is obtained.

[0089] According to embodiments of this disclosure, at least one directional fusion feature and an initial fusion feature can be processed based on an object detection algorithm to achieve object detection of at least one target object. Alternatively, at least one directional fusion feature can be processed based on an object detection algorithm to obtain an object detection result.

[0090] According to embodiments of this disclosure, by fusing image features, depth features, and normal vector features related to the target object, the resulting initial fused features can contain at least the normal vector information of the target object's surface, thereby enhancing the semantic information richness of the initial fused features. Then, feature information is extracted from the resulting initial fused features in at least one preset direction, which can make the resulting directional fused features more accurately represent the semantic information of the target object in the detection space corresponding to the preset direction. Furthermore, target object detection is performed on the target object based on the directional fused features, which can make the target object detection results more accurately represent the position of the target object in the detection space, thereby achieving the technical effect of improving the target object detection accuracy.

[0091] In one embodiment of this disclosure, the preset direction can be determined based on the arrangement of target objects in the image to be detected. For example, when multiple vehicles are driving according to road markings in the image to be detected, the preset direction and the direction of the road markings can be used as the preset direction, so that the obtained direction fusion feature can more accurately represent the position of the vehicle in the image to be detected.

[0092] Figure 3 A flowchart illustrating a target object detection method according to another embodiment of the present disclosure is shown.

[0093] According to embodiments of this disclosure, object depth information includes an object depth image, and object normal vector information includes an object normal vector image.

[0094] like Figure 3 As shown, the target object detection method may further include operations S310 to S320.

[0095] In operation S310, depth information detection is performed on the image to be detected to obtain the object depth image.

[0096] In operation S320, normal vector information is detected on the object depth image to obtain the object normal vector image.

[0097] According to embodiments of this disclosure, the image to be detected can be a monocular image. A monocular depth estimation algorithm can be used to detect depth information in the image to be detected, resulting in an object depth image. The depth map pixels in the obtained object depth image can represent the depth information of the corresponding image pixels in the image to be detected.

[0098] It should be understood that the monocular depth track estimation algorithm can perform feature extraction, depth classification, and depth regression on the image to be detected, and perform coordinate transformation based on the camera parameters of the image acquisition device that acquired the image to be detected, to obtain a pixel-level object depth image.

[0099] According to embodiments of this disclosure, an object depth image can be input into a normal vector information generation model to output an object normal vector image. The normal vector information generation model can be constructed based on the SNE (Stochastic Neighbor Embedding) model algorithm. By performing normal vector information detection on the generated object depth image, methods such as spatial variation relationships and depth plane fitting can be used to obtain a pixel-level object normal vector image. The normal vector pixels in the object normal vector image can contain pixel normal vector components with three dimensions (i.e., x, y, and z dimensions). These pixel normal vector components can accurately represent the surface image semantic information of the target object in the detection space, thereby enabling the subsequently obtained initial fusion features to help improve the detection accuracy of subsequent target object detection results.

[0100] According to embodiments of this disclosure, the target object detection method may further include the following operations.

[0101] The process involves acquiring point cloud data corresponding to the image to be detected, wherein the point cloud data is associated with at least one target object; performing depth information detection on the point cloud data to obtain object depth information; and performing normal vector information detection on the object depth information to obtain object normal vector information.

[0102] According to embodiments of this disclosure, the point cloud data to be detected can be obtained by detecting a target object in the detection space using a detection device such as a lidar. The point cloud data to be detected and the image to be detected can contain at least one target object.

[0103] According to embodiments of this disclosure, object depth information can be determined by determining the coordinate position of the point cloud data to be detected, and then object normal vector information can be determined by the same or corresponding algorithm as described above.

[0104] According to embodiments of this disclosure, by performing depth information detection on the point cloud data to be detected to obtain object depth information, the accuracy of object depth information can be further improved, thereby improving the accuracy of semantic information represented by the initial fused features, and thus achieving the technical effect of improving the accuracy of target object detection results.

[0105] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the image horizontal direction of the image to be detected, and the direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction.

[0106] In operation S230, extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature may include: inputting the initial fusion features into a first fusion feature extraction layer and outputting a first directional fusion feature; wherein the first fusion feature extraction layer is constructed based on a first convolutional kernel corresponding to the preset horizontal direction.

[0107] Figure 4A The schematic diagram illustrates the principle of extracting feature information of initial fusion features in at least one preset direction according to an embodiment of the present disclosure.

[0108] like Figure 4A As shown, the initial fusion feature 401 can be input to the first fusion feature extraction layer 410, and the output is a first-direction fusion feature 402. The first fusion feature extraction layer 410 can be constructed based on a convolutional neural network algorithm. Accordingly, the first fusion feature extraction layer 410 can be constructed based on a first convolutional kernel 411. The first convolutional kernel 411 can be a 1×5-dimensional convolutional kernel, so that the initial fusion feature 401 can be extracted in a preset horizontal direction through the first convolutional kernel 411 to obtain the first-direction fusion feature 402.

[0109] According to embodiments of this disclosure, by extracting the feature information of the initial fusion features in a preset horizontal direction, the spatial position information of image features, object normal vector features, and object depth features in the horizontal direction can be effectively decomposed, making the obtained first-direction fusion features more prominently represent the spatial position relationship of the target object in the horizontal direction in the detection space. Therefore, performing target object detection based on the first-direction fusion features can at least improve the accuracy of the target object detection results in representing the target object position in the preset horizontal direction.

[0110] According to embodiments of this disclosure, the preset direction includes a preset vertical direction corresponding to the vertical direction of the image to be detected, and the direction fusion feature includes a second direction fusion feature corresponding to the preset vertical direction.

[0111] In operation S230, extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature may include: inputting the initial fusion features into a second fusion feature extraction layer and outputting a second directional fusion feature; wherein the second fusion feature extraction layer is constructed based on a second convolution kernel corresponding to the preset vertical direction.

[0112] Figure 4B The schematic diagram illustrates the principle of extracting feature information of initial fusion features in at least one preset direction according to another embodiment of the present disclosure.

[0113] like Figure 4BAs shown, the initial fusion feature 401 can be input to the second fusion feature extraction layer 420, outputting a second-direction fusion feature 403. The second fusion feature extraction layer 420 can be constructed based on a convolutional neural network algorithm. Accordingly, the second fusion feature extraction layer 420 can be constructed based on a second convolutional kernel 421. The second convolutional kernel 421 can be a 5×1 dimensional convolutional kernel, thereby enabling feature extraction of the initial fusion feature 401 in a preset vertical direction through the second convolutional kernel 421, resulting in the second-direction fusion feature 403.

[0114] According to embodiments of this disclosure, by extracting the feature information of the initial fusion features in a preset vertical direction, the spatial position information of image features, object normal vector features, and object depth features in the scene depth direction of the detection space can be effectively decomposed. This makes the obtained second-direction fusion features more prominently represent the spatial positional relationship of the target object in the scene depth direction (or scene depth direction) of the detection space. Therefore, performing target object detection based on the second-direction fusion features can at least improve the accuracy of the target object detection results in representing the target object position in the scene direction.

[0115] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the horizontal direction of the image to be detected, and a preset vertical direction corresponding to the vertical direction of the image to be detected. The direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction and a second direction fusion feature corresponding to the preset vertical direction.

[0116] In operation S230, extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature may include: inputting the initial fusion features into a first fusion feature extraction layer and a second fusion feature extraction layer respectively, and outputting a first directional fusion feature and a second directional fusion feature; wherein, the first fusion feature extraction layer is constructed based on a first convolution kernel corresponding to a preset horizontal direction, and the second fusion feature extraction layer is constructed based on a second convolution kernel corresponding to a preset vertical direction.

[0117] According to embodiments of this disclosure, by extracting the first directional fusion feature and the second directional fusion feature respectively, the first directional fusion feature can more prominently characterize the horizontal spatial position of the target object in the detection space, and the second directional fusion feature can more prominently characterize the spatial position of the target object in the scene depth direction (or scene longitudinal direction) in the detection space.

[0118] It should be noted that the preset horizontal direction and preset vertical direction described in the embodiments of this disclosure can be the spatial horizontal direction and spatial vertical direction in the space to be detected, but are not limited to this. They can also be the horizontal direction relative to the image to be detected and the vertical direction relative to the image to be detected.

[0119] According to embodiments of this disclosure, in operation S230, target object detection is performed on at least one target object based on at least one directional fusion feature to obtain a target object detection result, which may include:

[0120] The first and second direction fusion features are fused based on the attention mechanism to obtain the target fusion features; and the target fusion features are input into the target object detection layer to output the target object detection results.

[0121] Figure 5 A schematic diagram of a target object detection model according to an embodiment of the present disclosure is shown.

[0122] like Figure 5 As shown, the target object detection model 500 may include: a depth detection layer 511, a normal vector detection layer 512, a first feature extraction layer 521, a second feature extraction layer 522, a third feature extraction layer 523, a first fusion layer 531, a first fusion feature extraction layer 541, a second fusion feature extraction layer 542, a second fusion layer 551, and a target object detection layer 561.

[0123] The image to be detected 501 can be an image acquired by a monocular image acquisition device installed on the unmanned vehicle. The monocular image acquisition device can be installed horizontally opposite to the unmanned vehicle to acquire the image to be monitored 501 with a large field of view, thereby improving the range of the unmanned vehicle's perception of the surrounding environment.

[0124] The image to be detected 501 can be input to the depth detection layer 511 of the target object detection model 500, and the output is an object depth image. The object depth image can be input to the normal vector detection layer 512, and the output is an object normal vector image.

[0125] The depth detection layer 511 can be constructed based on the BTS algorithm model, which includes an encoder with dense feature extraction and a decoder for predicting the required depth. The normal vector detection layer 512 can be constructed based on the SNE (Stochastic Neighbor Embedding) model algorithm.

[0126] The image to be detected 501 can be input to the first feature extraction layer 521, which outputs the image features of the image to be detected. The object depth image can be input to the second feature extraction layer 522, which outputs the depth features. The object normal vector image can be input to the third feature extraction layer 523, which outputs the normal vector features. The image features, depth features, and normal vector features can be input to the first fusion layer 531, which outputs the initial fused features.

[0127] According to embodiments of this disclosure, the first feature extraction layer 521, the second feature extraction layer 522, or the third feature extraction layer 523 can be neural network layers constructed based on the ResNet (Residual Networks) algorithm.

[0128] In one embodiment of this disclosure, the first fusion layer 531 can perform weighted processing based on the same image channels of the image to be detected, the object depth image, and the object normal vector image to obtain initial fusion features.

[0129] In one embodiment of this disclosure, the first fusion layer may be a neural network layer constructed based on a multilayer perceptron algorithm, thereby fully fusing image features, depth features, and normal vector features.

[0130] In one embodiment of this disclosure, the first fusion layer may also be an attention network layer constructed based on an attention network algorithm (such as the Transformer algorithm) to improve the accuracy of the depth feature information and normal vector feature information represented by the initial fusion features, thereby improving the subsequent target object detection accuracy.

[0131] The initial fusion features are input to the first fusion feature extraction layer 541 and the second fusion feature extraction layer 542, respectively, and output the first fusion feature corresponding to the preset horizontal direction and the second fusion feature corresponding to the preset vertical direction.

[0132] According to embodiments of this disclosure, the first fusion feature extraction layer 541 may include a first fusion feature extraction sub-layer constructed based on a 1×5 dimensional first convolutional kernel, and a first feature enhancement sub-layer constructed based on first attention weight parameters. The first feature enhancement sub-layer can enhance the spatial position characteristics of the first directional fusion features in the horizontal direction in the detection space.

[0133] According to embodiments of this disclosure, the second fusion feature extraction layer 542 may include a second fusion feature extraction sublayer constructed based on a 5×1 dimensional second convolutional kernel, and a second feature enhancement sublayer constructed based on second attention weight parameters. The second feature enhancement sublayer can enhance the spatial position characteristics of the second directional fusion features in the depth direction of the scene in the detection space.

[0134] It should be understood that the first attention weight parameter and the second attention weight parameter can be weight parameters obtained after training by the training method.

[0135] The first-direction fusion features and the second-direction fusion features can be input into the second fusion layer 551, which outputs the target fusion features. The second fusion layer 551 can be a neural network layer constructed based on the multilayer perceptron algorithm.

[0136] The target fusion features can be input into the target object detection layer 561 and output the target object detection result 502.

[0137] The target object detection layer 561 can be a target object detection head built based on a fully connected layer, and the target object detection result 502 output by the target object detection layer 561 can be a three-dimensional target detection box representing the target object.

[0138] Figure 6A A schematic diagram of an image to be detected according to an embodiment of the present disclosure is shown.

[0139] Figure 6B A schematic diagram of an object depth image according to an embodiment of the present disclosure is shown.

[0140] Figure 6C A schematic diagram of an object normal vector image according to an embodiment of the present disclosure is shown.

[0141] Figure 6D A schematic diagram illustrating the target object detection results according to an embodiment of the present disclosure is shown.

[0142] Combination Figures 6A to 6D As shown, the image to be detected 601 may contain a vehicle (target object). Processing the image to be detected 601 according to the target object detection method provided in this embodiment yields an object depth image 602 and an object normal vector image 603. By processing the image to be detected 601, the object depth image 602, and the object normal vector image 603 according to the target object detection method provided in this embodiment, the target object detection of the vehicle can be achieved, resulting in a target object detection result 604.

[0143] like Figure 6D As shown, the target object detection result 604 may contain a 3D detection box corresponding to the vehicle.

[0144] In one embodiment of this disclosure, the target object detection method provided in this disclosure can be applied in indoor scenes such as bank branches.

[0145] For example, an image acquisition device of a service consultation robot can be used to acquire images of the area within a bank branch. The images are then processed according to the target object detection method provided in this embodiment to obtain target object detection results. The service consultation robot can determine the location of users within the bank branch based on the target object detection results, thereby automatically avoiding users and preventing collisions. Alternatively, the target object detection results can be used to determine the location of users within the bank branch who require service consultation, thereby improving service efficiency.

[0146] Based on the above-described target object detection method, this disclosure also provides a target object detection device. The following will be combined with... Figure 8 The device is described in detail.

[0147] Figure 7 A schematic block diagram of a target object detection apparatus according to an embodiment of the present disclosure is shown.

[0148] like Figure 7 As shown, the target object device 700 of this embodiment includes a first feature extraction module 710, a fusion module 720, a second feature extraction module 730, and a target object detection module 740.

[0149] The first feature extraction module 710 is used to extract features from the image to be detected, the object depth information, and the object normal vector information respectively, to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object.

[0150] The fusion module 720 is used to fuse image features, depth features, and normal vector features to obtain initial fused features.

[0151] The second feature extraction module 730 is used to extract feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature.

[0152] The target object detection module 740 is used to perform target object detection on at least one target object based on features fused in at least one direction, and obtain the target object detection result.

[0153] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the image horizontal direction of the image to be detected, and the direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction.

[0154] The second feature extraction module includes a first-direction fusion feature acquisition unit.

[0155] The first directional fusion feature acquisition unit is used to input the initial fusion features into the first fusion feature extraction layer and output the first directional fusion features; wherein, the first fusion feature extraction layer is constructed based on the first convolution kernel corresponding to the preset horizontal direction.

[0156] According to embodiments of this disclosure, the preset direction includes a preset vertical direction corresponding to the vertical direction of the image to be detected, and the direction fusion feature includes a second direction fusion feature corresponding to the preset vertical direction.

[0157] The second feature extraction module includes a second-direction fusion feature acquisition unit.

[0158] The second directional fusion feature acquisition unit inputs the initial fusion features into the second fusion feature extraction layer and outputs the second directional fusion features; wherein, the second fusion feature extraction layer is constructed based on the second convolution kernel corresponding to the preset vertical direction.

[0159] According to embodiments of this disclosure, the preset direction includes a preset horizontal direction corresponding to the horizontal direction of the image to be detected, and a preset vertical direction corresponding to the vertical direction of the image to be detected. The direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction and a second direction fusion feature corresponding to the preset vertical direction.

[0160] The second feature extraction module includes a directional fusion feature acquisition unit.

[0161] The directional fusion feature acquisition unit is used to input the initial fusion features into the first fusion feature extraction layer and the second fusion feature extraction layer respectively, and output the first directional fusion feature and the second directional fusion feature; wherein, the first fusion feature extraction layer is constructed based on the first convolution kernel corresponding to the preset horizontal direction, and the second fusion feature extraction layer is constructed based on the second convolution kernel corresponding to the preset vertical direction.

[0162] According to embodiments of this disclosure, the target object detection module includes: a target fusion feature acquisition unit and a target object detection result acquisition unit.

[0163] The target fusion feature acquisition unit is used to fuse the first-direction fusion feature and the second-direction fusion feature based on the attention mechanism to obtain the target fusion feature.

[0164] The target object detection result acquisition unit is used to input the target fusion features into the target object detection layer and output the target object detection result.

[0165] According to embodiments of this disclosure, object depth information includes an object depth image, and object normal vector information includes an object normal vector image.

[0166] The target object detection device also includes: a first depth information detection module and a first normal vector information detection module.

[0167] The first depth information detection module is used to perform depth information detection on the image to be detected, and obtain the object depth image.

[0168] The first normal vector information detection module is used to detect the normal vector information of the object depth image to obtain the object normal vector image.

[0169] According to embodiments of this disclosure, the target object detection device further includes: a point cloud data acquisition module, a second depth information detection module, and a second normal vector information detection module.

[0170] The point cloud data acquisition module is used to acquire point cloud data corresponding to the image to be detected, and the point cloud data to be detected is associated with at least one target object.

[0171] The second depth information detection module is used to perform depth information detection on the point cloud data to be detected, and obtain the object depth information.

[0172] The second normal vector information detection module is used to detect the normal vector information of the object depth information to obtain the object normal vector information.

[0173] According to embodiments of this disclosure, any plurality of modules among the first feature extraction module 710, fusion module 720, second feature extraction module 730, and target object detection module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first feature extraction module 710, fusion module 720, second feature extraction module 730, and target object detection module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first feature extraction module 710, the fusion module 720, the second feature extraction module 730, and the target object detection module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0174] Figure 8A block diagram schematically illustrates an electronic device suitable for implementing a target object detection method according to an embodiment of the present disclosure.

[0175] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0176] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0177] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0178] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0179] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0180] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0181] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0182] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0183] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0184] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0187] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A target object detection method, comprising: Feature extraction is performed on the image to be detected, the object depth information, and the object normal vector information to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object; The image features, depth features, and normal vector features are fused to obtain the initial fused features; Extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature, including: constructing a fusion feature extraction layer based on a convolutional kernel corresponding to the preset direction, and extracting feature information of the initial fusion features in the corresponding preset direction through the fusion feature extraction layer; and Based on at least one of the directional fusion features, target object detection is performed on the at least one target object to obtain a target object detection result.

2. The method according to claim 1, wherein, The preset direction includes a preset horizontal direction corresponding to the image horizontal direction of the image to be detected, and the direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction; The step of extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes: The initial fusion features are input into the first fusion feature extraction layer, and the first direction fusion features are output. The first fusion feature extraction layer is constructed based on a first convolutional kernel corresponding to the preset horizontal direction.

3. The method according to claim 1, wherein, The preset direction includes a preset vertical direction corresponding to the vertical direction of the image to be detected, and the direction fusion feature includes a second direction fusion feature corresponding to the preset vertical direction; The step of extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes: The initial fusion features are input into the second fusion feature extraction layer, and the second direction fusion features are output. The second fusion feature extraction layer is constructed based on a second convolutional kernel corresponding to the preset vertical direction.

4. The method according to claim 1, wherein, The preset direction includes a preset horizontal direction corresponding to the horizontal direction of the image to be detected, and a preset vertical direction corresponding to the vertical direction of the image to be detected. The direction fusion feature includes a first direction fusion feature corresponding to the preset horizontal direction, and a second direction fusion feature corresponding to the preset vertical direction. The step of extracting feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature includes: The initial fusion features are respectively input into the first fusion feature extraction layer and the second fusion feature extraction layer, and the first directional fusion features and the second directional fusion features are output. The first fusion feature extraction layer is constructed based on a first convolutional kernel corresponding to the preset horizontal direction, and the second fusion feature extraction layer is constructed based on a second convolutional kernel corresponding to the preset vertical direction.

5. The method according to claim 4, wherein, The step of performing target object detection on the at least one target object based on at least one of the directional fusion features to obtain the target object detection result includes: Based on an attention mechanism, the first-direction fusion feature and the second-direction fusion feature are fused to obtain the target fusion feature; and The target fusion features are input into the target object detection layer, and the target object detection result is output.

6. The method according to claim 1, wherein, The object depth information includes an object depth image, and the object normal vector information includes an object normal vector image; The target object detection method further includes: Depth information detection is performed on the image to be detected to obtain the object depth image; Normal vector information is detected on the depth image of the object to obtain the object normal vector image.

7. The method according to claim 1, further comprising: Acquire point cloud data corresponding to the image to be detected, wherein the point cloud data to be detected is associated with at least one of the target objects; Depth information detection is performed on the point cloud data to be detected to obtain the object depth information; as well as The object's depth information is analyzed by normal vector detection to obtain the object's normal vector information.

8. A target object detection device, comprising: The first feature extraction module is used to extract features from the image to be detected, the object depth information, and the object normal vector information respectively, to obtain image features, depth features, and normal vector features, wherein the image to be detected, the object depth information, and the object normal vector information are all associated with at least one target object; The fusion module is used to fuse the image features, the depth features, and the normal vector features to obtain initial fused features; The second feature extraction module is used to extract feature information of the initial fusion features in at least one preset direction to obtain at least one directional fusion feature, including: constructing a fusion feature extraction layer based on a convolutional kernel corresponding to the preset direction, and extracting feature information of the initial fusion features in the corresponding preset direction through the fusion feature extraction layer; and The target object detection module is used to perform target object detection on the at least one target object based on at least one directional fusion feature, and obtain a target object detection result.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images