A detection target reconstruction method, system, terminal and medium

Abstract

The present invention discloses a method, system, terminal, and medium for reconstructing a detected target. The method includes: obtaining an original image and text prompt information of the detected target; fusing the original image and text prompt information to obtain multimodal features; obtaining a detection frame of the detected target, a category confidence of the detected target, and a query feature of the detected target through a detector based on the multimodal features; determining visual features corresponding to the original image, and obtaining detection features through a detector based on the visual features and the multimodal features; and obtaining two-dimensional key point coordinates of the detected target, three-dimensional key point coordinates of the detected target, and camera position coordinates through a decoder based on the detection frame of the detected target, the category confidence of the detected target, the query feature of the detected target, and the detection features. The present invention avoids the problem of repeated feature extraction, significantly improves computational efficiency, eliminates the disadvantage of gradual error accumulation, and improves the accuracy and robustness of the reconstruction results.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a detection target reconstruction method, system, terminal and medium. Background Art

[0002] Currently, 3D reconstruction has garnered widespread attention in the field of computer vision, particularly in applications such as human-body interaction, augmented reality, and robotics. Existing 3D reconstruction methods typically employ a multi-stage process involving multiple independent steps, including object detection, classification, and pose estimation. This multi-stage process suffers from repeated feature extraction, leading to redundant computations and accumulated errors, thus limiting its effectiveness and efficiency in practical applications. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a detection target reconstruction method, system, terminal and medium in response to the above-mentioned defects of the prior art, aiming to solve the problem that the multi-stage three-dimensional reconstruction method of the prior art is prone to redundant calculations and cumulative errors, thereby limiting the effect and efficiency in practical applications.

[0004] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0005] In a first aspect, the present invention provides a detection target reconstruction method, wherein the method comprises:

[0006] Obtaining an original image and text prompt information of the detection target, and fusing the original image and text prompt information to obtain a multimodal feature;

[0007] Based on the multimodal features, a detector is used to obtain a detection frame of the detection target, a category confidence of the detection target, and a query feature of the detection target;

[0008] Determining visual features corresponding to the original image, and obtaining detection features through a detector based on the visual features and the multimodal features;

[0009] According to the detection frame of the detection target, the category confidence of the detection target, the query features of the detection target and the detection features, the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates are obtained through the decoder.

[0010] In one implementation, fusing the original image with the text prompt information to obtain a multimodal feature includes:

[0011] The detector includes a backbone network and an encoder;

[0012] Based on the backbone network, feature extraction is performed on the original image and the text prompt information respectively to obtain image features and text features;

[0013] The image features and the text features are fused through the encoder to obtain the multimodal features.

[0014] In one implementation, obtaining a detection frame of a detection target, a category confidence of the detection target, and a query feature of the detection target by a detector based on the multimodal feature includes:

[0015] The detector further includes an object detection decoder and a first multi-layer perceptron;

[0016] Determining, based on a first attention mechanism of the object detection decoder, a query feature of the detection target corresponding to the multimodal feature;

[0017] The query features of the detection target are predicted based on the first multi-layer perceptron to obtain a detection frame of the detection target and a category confidence of the detection target.

[0018] In one implementation, determining the visual features corresponding to the original image and obtaining a detection feature through a detector based on the visual features and the multimodal features includes:

[0019] The detector also includes a convolutional network;

[0020] Determining visual features corresponding to the original image based on the backbone network;

[0021] Inputting the visual features into the convolutional network to obtain convolutional features;

[0022] The convolution feature is cascaded with the multimodal feature to obtain the detection feature.

[0023] In one implementation, obtaining the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates through a decoder based on the detection frame of the detection target, the category confidence of the detection target, the query feature of the detection target, and the detection feature includes:

[0024] Filtering the query features of the detection target to obtain positive sample query features;

[0025] Expanding and updating the positive sample query feature and the detection frame of the detection target to obtain an updated two-dimensional key point query feature and an updated two-dimensional key point detection frame;

[0026] Based on the updated two-dimensional key point query feature and the updated two-dimensional key point detection frame, a three-dimensional key point query feature, a three-dimensional key point detection frame, a camera position query feature, and a camera position detection frame are obtained;

[0027] Based on the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the three-dimensional key point detection frame, the camera position query feature, the camera position detection frame and the detection feature, the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates are obtained.

[0028] In one implementation, filtering the query features of the detection target to obtain positive sample query features includes:

[0029] Matching the query feature of the detection target, the detection frame of the detection target, and the category confidence of the detection target to obtain the positive sample query feature;

[0030] or,

[0031] The category confidence of the detection target is screened according to the query feature of the preset confidence threshold to obtain the positive sample query feature.

[0032] In one implementation, the positive sample query feature and the detection frame of the detection target are expanded and updated to obtain an updated two-dimensional key point query feature and an updated two-dimensional key point detection frame, including:

[0033] The decoder includes a second multi-layer perceptron, a third multi-layer perceptron, and a second attention mechanism;

[0034] Constructing the positive sample query feature and the detection frame of the detection target according to the number of two-dimensional key points of the detection target to obtain an expanded query feature and an expanded detection frame;

[0035] Initialize each 2D keypoint into a learnable vector representation;

[0036] Obtaining a two-dimensional key point query feature of the detection target by fusing the learnable vector representation with the extended query feature;

[0037] Updating the extended detection frame based on a second multi-layer perceptron to obtain a two-dimensional key point detection frame;

[0038] The key point query feature and the two-dimensional key point detection frame are iteratively updated through the second attention mechanism and the third multi-layer perceptron to obtain updated two-dimensional key point query features and updated two-dimensional key point detection frame.

[0039] In one implementation, obtaining a 3D key point query feature, a 3D key point detection frame, a camera position query feature, and a camera position detection frame based on the updated 2D key point query feature and the updated 2D key point detection frame includes:

[0040] Obtaining a preset dimension-raising matrix, where the dimension-raising matrix reflects the linear combination weights of two-dimensional elements and three-dimensional elements;

[0041] The updated two-dimensional key point query features and the updated two-dimensional key point detection frame are calculated based on the dimension-raising matrix to obtain three-dimensional key point query features, camera position query features, three-dimensional key point detection frame, and camera position detection frame.

[0042] In one implementation, obtaining the two-dimensional key point coordinates, the three-dimensional key point coordinates, and the camera position coordinates of the detection target based on the updated two-dimensional key point query feature, the three-dimensional key point query feature, the camera position query feature, the updated two-dimensional key point detection frame, the three-dimensional key point detection frame, the camera position detection frame, and the detection feature includes:

[0043] The decoder further includes a third attention mechanism, a fourth multi-layer perceptron, and a fifth multi-layer perceptron;

[0044] After the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the camera position query feature, the three-dimensional key point detection frame, and the camera position detection frame are adjusted by the third attention mechanism, the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the camera position query feature, the three-dimensional key point detection frame, and the camera position detection frame are iteratively updated with the detection feature in sequence by the third attention mechanism and the fourth multi-layer perceptron to obtain a further updated two-dimensional key point query feature, a further updated two-dimensional key point detection frame, an updated three-dimensional key point query feature, an updated camera position query feature, an updated three-dimensional key point detection frame, and an updated camera position detection frame;

[0045] The re-updated two-dimensional key point query feature, the updated three-dimensional key point query feature, the updated camera position query feature, the re-updated two-dimensional key point detection frame, the updated three-dimensional key point detection frame, and the updated camera position detection frame are passed through a fifth multi-layer perceptron to obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates.

[0046] In a second aspect, an embodiment of the present invention further provides a detection target reconstruction system, wherein the system includes:

[0047] A multimodal feature fusion module is used to obtain the original image and text prompt information of the detection target and fuse the original image and text prompt information to obtain multimodal features;

[0048] A multimodal detection module, configured to obtain a detection frame of a detection target, a category confidence of the detection target, and a query feature of the detection target through a detector based on the multimodal features;

[0049] a detection feature determination module, configured to determine visual features corresponding to the original image, and obtain detection features through a detector based on the visual features and the multimodal features;

[0050] The detection target reconstruction module is used to obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates based on the detection frame of the detection target, the category confidence of the detection target, the query features of the detection target and the detection features through the decoder.

[0051] In a third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a detection target reconstruction program stored in the memory and runnable on the processor. When the processor executes the detection target reconstruction program, the steps of the detection target reconstruction method of any one of the above-mentioned schemes are implemented.

[0052] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a detection target reconstruction program is stored on the computer-readable storage medium, and when the detection target reconstruction program is executed by a processor, the steps of the detection target reconstruction method described in any one of the above schemes are implemented.

[0053] Beneficial effects: Compared with the prior art, the present invention provides a method for reconstructing a detected target. The present invention first obtains the original image and text prompt information of the detected target. Then, the original image and text prompt information are fused to obtain multimodal features. Next, based on the multimodal features, a detector is used to obtain a detection frame of the detected target, the category confidence of the detected target, and the query features of the detected target. Then, the visual features corresponding to the original image are determined, and detection features are obtained by the detector based on the visual features and the multimodal features. Finally, based on the detection frame of the detected target, the category confidence of the detected target, the query features of the detected target, and the detection features, a decoder is used to obtain the two-dimensional key point coordinates of the detected target, the three-dimensional key point coordinates of the detected target, and the camera position coordinates. Compared with the traditional multi-stage detection target reconstruction method, the present invention only inputs the original image and text prompt information once, avoiding complex intermediate steps. It also avoids the problem of repeated feature extraction caused by multi-stage processing in traditional methods. In addition, the extracted features of the present invention can be reused, thereby greatly improving computational efficiency. In addition, the present invention eliminates the drawback of the gradual accumulation of errors in each stage of the multi-stage method, and improves the accuracy and robustness of the reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention provides a flowchart of a preferred embodiment of the detection target reconstruction method.

[0055] Figure 2 This is a logical framework diagram of the detection target reconstruction method provided in an embodiment of the present invention.

[0056] Figure 3 for Figure 2 The interactive module architecture diagram in .

[0057] Figure 4 A schematic diagram of the architecture of a detection target reconstruction system provided by an embodiment of the present invention.

[0058] Figure 5 This is a functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents, operations, or steps, nor must they be executed in the order described. For example, some operations or steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0061] It should be understood that the terms used in this specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0062] It should be understood that, to facilitate a clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the first control information and the second control information are merely used to distinguish different control information and do not limit their order.

[0063] Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0064] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] Existing methods for detecting and reconstructing objects typically employ a multi-stage process, encompassing multiple independent steps, including object detection, object classification, and pose estimation. However, this multi-stage approach can lead to several issues. For example, because each stage is performed independently, the system must perform feature calculations for each step separately, which introduces redundant computational burdens. For example, both object detection and pose estimation may require feature extraction of the hand region, but these features are repeatedly calculated across different stages, increasing computational complexity and time overhead. Furthermore, a multi-stage approach cannot effectively eliminate the accumulated errors between stages. The output of each stage serves as the input for the next stage, but if an error exists in the output of a stage, this error will gradually accumulate in subsequent stages, reducing the accuracy of the final result. For example, the object detection stage may misjudge the position of the hand, and this error will be propagated to the subsequent left and right hand classification and pose estimation, ultimately affecting the quality of the 3D hand reconstruction.

[0066] To this end, this embodiment provides a method for reconstructing a detected target. This method avoids complex intermediate steps and the repeated feature extraction issues associated with multi-stage processing in traditional methods. The extracted features can be reused, significantly improving computational efficiency. Furthermore, the drawback of the gradual accumulation of errors at each stage in multi-stage methods is eliminated, improving the accuracy and robustness of the reconstruction results. In specific applications, the original image and textual prompt information of the detected target are first acquired. The original image and textual prompt information are then fused to obtain multimodal features. Next, a detector is used to obtain a detection bounding box, a category confidence score, and a query feature of the detected target based on the multimodal features. Next, visual features corresponding to the original image are determined, and detection features are obtained using a detector based on the visual features and the multimodal features. Finally, a decoder is used to obtain the two-dimensional keypoint coordinates, three-dimensional keypoint coordinates, and camera position coordinates of the detected target based on the detection bounding box, the category confidence score, the query feature, and the detected features.

[0067] The detection target reconstruction method of this embodiment can be applied to a terminal, which is an intelligent product terminal such as a computer, a mobile phone, and a smart TV. Figure 1 As shown in , the detection target reconstruction method of this embodiment includes the following steps:

[0068] Step S100: obtaining an original image and text prompt information of a detection target, and fusing the original image and text prompt information to obtain a multimodal feature.

[0069] Combine Figure 2 As shown in , this embodiment first uses a detector to detect the detection target in the original image and the text prompt information, and determines the detection frame of the detection target, the category confidence of the detection target, and the query feature (Query) of the detection target. The text prompt information of this embodiment is used to prompt the detection target in the original image. For example, if the original image is a hand image, the corresponding text prompt information can be the word "hand". Specifically, the detector of this embodiment is the Grounding DINO detector, which is a multimodal detection framework. Specifically combined with Figure 2As shown in , the detector includes: a backbone network (specifically including a text backbone network and a visual backbone network), an encoder, a target detection decoder, a convolutional network and a detection module, wherein the detection module includes a first multi-layer perceptron. When the original image and text prompt information are input, the visual backbone network and the text backbone network in the detector will respectively extract features of the original image and the text prompt information to obtain image features and text features. Then, this embodiment can fuse the image features and the text features based on the encoder in the detector to obtain the multimodal features. This embodiment obtains multimodal features by fusing image features and text features, so that the accuracy of reconstruction can be improved in the subsequent reconstruction process.

[0070] Step S200: obtaining a detection frame of a detection target, a category confidence of the detection target, and a query feature of the detection target through a detector based on the multimodal features.

[0071] Furthermore, this embodiment inputs the multimodal features into the target detection decoder of the detector, and determines the query feature (Query) of the detection target corresponding to the multimodal features based on the first attention mechanism (including the self-attention mechanism and the deformable attention mechanism) of the target detection decoder. The query feature of the detection target obtained at this time is Figure 2 The query feature of the detection target is then input into the detection module of the detector. The query feature of the detection target is predicted based on the first multi-layer perceptron in the detection module to obtain the detection box of the detection target and the category confidence of the detection target.

[0072] Step S300: Determine the visual features corresponding to the original image, and obtain detection features through a detector based on the visual features and the multimodal features.

[0073] Combine Figure 2 As shown, the visual backbone network of the detector of this embodiment extracts features from the original image to obtain visual features of the original image, and then inputs the visual features into the convolutional network of the detector to obtain convolution features. Next, the convolution features are cascaded with the multimodal features to obtain the detection features. This embodiment determines the detection features by feature cascade, which provides a three-dimensional basis for the subsequent reconstruction of the detection target, so as to better perform three-dimensional reconstruction and improve the accuracy of the detection target reconstruction.

[0074] Step S400: Obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates through a decoder based on the detection frame of the detection target, the category confidence of the detection target, the query features of the detection target, and the detection features.

[0075] After obtaining the detection frame of the detection target, the category confidence of the detection target, the query feature of the detection target and the detection feature, this embodiment inputs the detection frame of the detection target, the category confidence of the detection target, the query feature of the detection target and the detection feature into the preset decoding, Figure 2 It can be seen from the figure that the decoder at this time can be a two-dimensional-three-dimensional interactive decoder. The specific architecture of the two-dimensional-three-dimensional interactive decoder is as follows: Figure 2 As shown on the right side of the figure, based on the 2D-3D interactive decoder, the 2D key point coordinates of the detection target, the 3D key point coordinates of the detection target and the camera position coordinates are estimated, thereby realizing the 3D reconstruction of the detection target.

[0076] In specific applications, since the query features of the detection target are not all positive samples, this embodiment first filters the query features of the detection target to obtain the positive sample query features. When filtering, this embodiment inputs the query features (Query) of the detection target and the detection frame of the detection target and the category confidence of the detection target obtained in the above step S100, and the output is the positive sample query features. This embodiment can obtain the positive sample query features based on a matching method (such as Hungarian matching), that is, matching the query features of the detection target, the detection frame of the detection target and the category confidence of the detection target to obtain the positive sample query features. Alternatively, in other implementations, this embodiment can filter the category confidence of the detection target according to the query features of a preset confidence threshold to obtain the positive sample query features. Specifically, the positive sample query features can be obtained by filtering out the category confidence that is higher than the confidence threshold from the category confidence of the detection target.

[0077] Furthermore, since the positive sample query feature can only represent the target instance, it does not contain the key point information of the detection target. For this reason, after obtaining the positive sample query feature, the present embodiment expands the positive sample query feature and the detection frame of the detection target to obtain the two-dimensional key point query feature and the two-dimensional key point detection frame of the detection target. The present embodiment can construct the positive sample query feature and the detection frame of the detection target respectively according to the number of two-dimensional key points of the detection target to obtain the extended query feature and the extended detection frame. Specifically, the present embodiment copies the positive sample query feature according to the number of two-dimensional key points of the detection target, for example, the positive sample query feature is copied K times, K is the number of two-dimensional key points of the detection target, to obtain the extended query feature. Each two-dimensional key point is then initialized to a learnable vector representation, and the two-dimensional key point query feature of the detection target is obtained by fusing the learnable vector representation with the extended query feature, for example, adding the learnable vector representation to the extended query feature. Similarly, the present embodiment can copy the detection frame of the detection target K times, K is the number of two-dimensional key points of the detection target, to obtain the extended detection frame. Then, the present embodiment can be based on the decoder (i.e. Figure 2 The second multi-layer perceptron in the 2D-3D interactive decoder (in the 2D-3D interactive decoder) updates the expanded detection box to obtain a 2D keypoint detection box. Through these steps, both the query features and the detection box reflect the 2D keypoints of the detected object, facilitating 3D reconstruction of the detected object in subsequent steps.

[0078] Furthermore, in this embodiment, the key point query feature and the two-dimensional key point detection frame are sequentially input into Figure 2 The 2D module of the 2D-3D interactive decoder includes a second attention mechanism (self-attention mechanism and deformable attention mechanism) and a third multi-layer perceptron. Therefore, the keypoint query features and the 2D keypoint detection boxes are iteratively updated through the self-attention mechanism, the deformable attention mechanism, and the third multi-layer perceptron, respectively, to obtain updated 2D keypoint query features and updated 2D keypoint detection boxes.

[0079] Furthermore, this embodiment can obtain three-dimensional key point query features, camera position query features, three-dimensional key point detection frames, and camera position detection frames based on the updated two-dimensional key point query features and the updated two-dimensional key point detection frames, thereby completing two-dimensional to three-dimensional reconstruction. Specifically, this embodiment pre-sets a learnable dimension-raising matrix that reflects the linear combination weights of two-dimensional elements and three-dimensional elements. Then, based on the dimension-raising matrix, the updated two-dimensional key point query features and the updated two-dimensional key point detection frames are calculated to obtain three-dimensional key point query features, camera position query features, three-dimensional key point detection frames, and camera position detection frames. Specifically, this embodiment multiplies the updated two-dimensional key point query features with the dimension-raising matrix to obtain three-dimensional key point query features and camera position query features. Similarly, the updated two-dimensional key point detection frame is multiplied with the dimension-raising matrix to obtain three-dimensional key point detection frames and camera position detection frames. In addition, preferably, this embodiment can also input the obtained three-dimensional key point detection frame and camera position detection frame into a multi-layer perceptron, and adjust the multi-layer perceptron to better represent the three-dimensional key point detection frame and camera position detection frame.

[0080] Furthermore, this embodiment can obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates based on the updated two-dimensional key point query features, three-dimensional key point query features, camera position query features, updated two-dimensional key point detection frame, three-dimensional key point detection frame, camera position detection frame and detection features. Specifically, combined with Figure 3 As shown, this embodiment firstly based on the decoder (ie Figure 2 The interactive module in the 2D-3D interactive decoder in the image is used to process the updated 2D key point query features, 3D key point query features, camera position query features, updated 2D key point detection frames, 3D key point detection frames, camera position detection frames and detection features. The interactive module includes a third attention mechanism (including a self-attention mechanism and a deformable attention mechanism) and a fourth multi-layer perceptron. Combined with Figure 3As shown, this embodiment first adjusts the updated 2D key point query features, 3D key point query features, camera position query features, updated 2D key point detection frames, 3D key point detection frames, and camera position detection frames through the self-attention mechanism of the third attention mechanism. These features are then iteratively updated with the detection features through the deformable attention mechanism of the third attention mechanism and the fourth multi-layer perceptron, resulting in further updated 2D key point query features, updated 3D key point query features, updated camera position query features, further updated 2D key point detection frames, updated 3D key point detection frames, and updated camera position detection frames. This embodiment utilizes the self-attention mechanism and deformable attention mechanism to implement the above iterative update process, ensuring effective 2D and 3D joint learning.

[0081] Furthermore, this embodiment can input the updated two-dimensional key point query feature, the updated three-dimensional key point query feature, the updated camera position query feature, the updated two-dimensional key point detection frame, the updated three-dimensional key point detection frame, and the updated camera position detection frame into the image. Figure 2 The decoder (i.e. Figure 2 The key point module in the 2D-3D interactive decoder in the embodiment of the present invention comprises a fifth multi-layer perceptron. After processing by the fifth multi-layer perceptron, the 2D key point coordinates of the detection target, the 3D key point coordinates of the detection target and the camera position coordinates can be obtained, thereby realizing 3D reconstruction of the detection target. Figure 2 As shown in the rightmost figure.

[0082] In summary, the present embodiment provides a single-stage detection target reconstruction method. Compared with the traditional multi-stage detection target reconstruction method, the original image and text prompt information are only input once in the present embodiment, which avoids complex intermediate steps and also avoids the problem of repeated feature extraction caused by multi-stage processing in the traditional method. The extracted features of the present embodiment can be reused, thereby greatly improving the computational efficiency. In addition, the present embodiment eliminates the disadvantage of the gradual accumulation of errors in each stage of the multi-stage method, and improves the accuracy and robustness of the reconstruction results. The method of the present embodiment is simpler, the framework design is more concise, and complex intermediate steps and dependencies are avoided, making the method easier to implement and apply in actual scenarios. Whether it is reconstructing the three-dimensional structure of the detection target in a dynamic environment or processing the changes in the posture of the detection target in a complex background, the present invention can show high adaptability and reliability.

[0083] Based on the above embodiments, the present invention also provides a detection target reconstruction system, such as Figure 4As shown in , the system of this embodiment includes: a multimodal feature fusion module 10, a multimodal detection module 20, a detection feature determination module 30, and a detection target reconstruction module 40. Specifically, the multimodal feature fusion module 10 of this embodiment is used to obtain the original image and text prompt information of the detection target, and fuse the original image and text prompt information to obtain multimodal features. The multimodal detection module 20 is used to obtain the detection frame of the detection target, the category confidence of the detection target, and the query feature of the detection target through a detector based on the multimodal features. The detection feature determination module 30 is used to determine the visual features corresponding to the original image, and obtain the detection features through a detector based on the visual features and the multimodal features. The detection target reconstruction module 40 is used to obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates according to the detection frame of the detection target, the category confidence of the detection target, the query feature of the detection target, and the detection features through a decoder.

[0084] The working principles of each module in the detection target reconstruction system of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0085] Each module in the above-mentioned detection target reconstruction system can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the terminal in hardware form, or can be stored in the memory in the terminal in software form, so that the processor can call and execute the corresponding operations of each module.

[0086] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 5 The terminal may include one or more processors 100 ( Figure 5 (only one is shown), memory 101, and a computer program 102 stored in memory 101 and executable on one or more processors 100, for example, a sleep analysis program based on multi-sensor data. When one or more processors 100 execute computer program 102, they can implement the various steps of an embodiment of a sleep analysis method based on multi-sensor data. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in an embodiment of a sleep analysis system based on multi-sensor data, without limitation herein.

[0087] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory 101 may include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is about to be output.

[0089] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0090] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A detection target reconstruction method, characterized in that: The method comprises: Obtaining an original image and text prompt information of the detection target, and fusing the original image and text prompt information to obtain a multimodal feature; Based on the multimodal features, a detector is used to obtain a detection frame of the detection target, a category confidence of the detection target, and a query feature of the detection target; Determining visual features corresponding to the original image, and obtaining detection features through a detector based on the visual features and the multimodal features; Obtaining the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates through a decoder according to the detection frame of the detection target, the category confidence of the detection target, the query features of the detection target, and the detection features; According to the detection frame of the detection target, the category confidence of the detection target, the query feature of the detection target, and the detection feature, the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates are obtained through a decoder, including: Filtering the query features of the detection target to obtain positive sample query features; Expanding and updating the positive sample query feature and the detection frame of the detection target to obtain an updated two-dimensional key point query feature and an updated two-dimensional key point detection frame; Based on the updated two-dimensional key point query feature and the updated two-dimensional key point detection frame, a three-dimensional key point query feature, a three-dimensional key point detection frame, a camera position query feature, and a camera position detection frame are obtained; Based on the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the three-dimensional key point detection frame, the camera position query feature, the camera position detection frame and the detection feature, the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates are obtained.

2. The detection target reconstruction method according to claim 1, characterized in that: The original image and the text prompt information are fused to obtain multimodal features, including: The detector includes a backbone network and an encoder; Based on the backbone network, feature extraction is performed on the original image and the text prompt information respectively to obtain image features and text features; The image features and the text features are fused through the encoder to obtain the multimodal features.

3. The detection target reconstruction method according to claim 1, characterized in that: Obtaining a detection frame of a detection target, a category confidence of the detection target, and a query feature of the detection target through a detector based on the multimodal features, including: The detector further includes an object detection decoder and a first multi-layer perceptron; Determining, based on a first attention mechanism of the object detection decoder, a query feature of the detection target corresponding to the multimodal feature; The query features of the detection target are predicted based on the first multi-layer perceptron to obtain a detection frame of the detection target and a category confidence of the detection target.

4. The detection target reconstruction method according to claim 2, characterized in that: Determining a visual feature corresponding to the original image, and obtaining a detection feature through a detector based on the visual feature and the multimodal feature, including: The detector also includes a convolutional network; Determining visual features corresponding to the original image based on the backbone network; Inputting the visual features into the convolutional network to obtain convolutional features; The convolution feature is cascaded with the multimodal feature to obtain the detection feature.

5. The detection target reconstruction method according to claim 1, characterized in that: Filter the query features of the detection target to obtain positive sample query features, including: Matching the query feature of the detection target, the detection frame of the detection target, and the category confidence of the detection target to obtain the positive sample query feature; or, The category confidence of the detection target is screened according to the query feature of the preset confidence threshold to obtain the positive sample query feature.

6. The detection target reconstruction method according to claim 1, characterized in that: Expanding and updating the positive sample query feature and the detection frame of the detection target to obtain an updated two-dimensional key point query feature and an updated two-dimensional key point detection frame, including: The decoder includes a second multi-layer perceptron, a third multi-layer perceptron, and a second attention mechanism; Constructing the positive sample query feature and the detection frame of the detection target according to the number of two-dimensional key points of the detection target to obtain an expanded query feature and an expanded detection frame; Initialize each 2D keypoint into a learnable vector representation; Obtaining a two-dimensional key point query feature of the detection target by fusing the learnable vector representation with the extended query feature; Updating the extended detection frame based on a second multi-layer perceptron to obtain a two-dimensional key point detection frame; The key point query feature and the two-dimensional key point detection frame are iteratively updated through the second attention mechanism and the third multi-layer perceptron to obtain updated two-dimensional key point query features and updated two-dimensional key point detection frame.

7. The detection target reconstruction method according to claim 1, characterized in that: Based on the updated two-dimensional key point query feature and the updated two-dimensional key point detection frame, a three-dimensional key point query feature, a three-dimensional key point detection frame, a camera position query feature, and a camera position detection frame are obtained, including: Obtaining a preset dimension-raising matrix, where the dimension-raising matrix reflects the linear combination weights of two-dimensional elements and three-dimensional elements; The updated two-dimensional key point query features and the updated two-dimensional key point detection frame are calculated based on the dimension-raising matrix to obtain three-dimensional key point query features, camera position query features, three-dimensional key point detection frame, and camera position detection frame.

8. The detection target reconstruction method according to claim 6 or 7, characterized in that: Obtaining the two-dimensional key point coordinates, the three-dimensional key point coordinates, and the camera position coordinates of the detection target based on the updated two-dimensional key point query feature, the three-dimensional key point query feature, the camera position query feature, the updated two-dimensional key point detection frame, the three-dimensional key point detection frame, the camera position detection frame, and the detection feature, including: The decoder further includes a third attention mechanism, a fourth multi-layer perceptron, and a fifth multi-layer perceptron; After the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the camera position query feature, the three-dimensional key point detection frame, and the camera position detection frame are adjusted by the third attention mechanism, the updated two-dimensional key point query feature, the updated two-dimensional key point detection frame, the three-dimensional key point query feature, the camera position query feature, the three-dimensional key point detection frame, and the camera position detection frame are iteratively updated with the detection feature in sequence by the third attention mechanism and the fourth multi-layer perceptron to obtain a further updated two-dimensional key point query feature, a further updated two-dimensional key point detection frame, an updated three-dimensional key point query feature, an updated camera position query feature, an updated three-dimensional key point detection frame, and an updated camera position detection frame; The re-updated two-dimensional key point query feature, the updated three-dimensional key point query feature, the updated camera position query feature, the re-updated two-dimensional key point detection frame, the updated three-dimensional key point detection frame, and the updated camera position detection frame are passed through a fifth multi-layer perceptron to obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target, and the camera position coordinates.

9. A detection target reconstruction system, characterized in that: The system is used to implement the steps of the detection target reconstruction method according to any one of claims 1 to 8, and the system includes: A multimodal feature fusion module is used to obtain the original image and text prompt information of the detection target and fuse the original image and text prompt information to obtain multimodal features; A multimodal detection module, configured to obtain a detection frame of a detection target, a category confidence of the detection target, and a query feature of the detection target through a detector based on the multimodal features; a detection feature determination module, configured to determine visual features corresponding to the original image, and obtain detection features through a detector based on the visual features and the multimodal features; The detection target reconstruction module is used to obtain the two-dimensional key point coordinates of the detection target, the three-dimensional key point coordinates of the detection target and the camera position coordinates based on the detection frame of the detection target, the category confidence of the detection target, the query features of the detection target and the detection features through the decoder.

10. A terminal, characterized in that: The terminal includes a memory, a processor, and a detection target reconstruction program stored in the memory and executable on the processor. When the processor executes the detection target reconstruction program, the steps of the detection target reconstruction method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a detection target reconstruction program, and when the detection target reconstruction program is executed by the processor, the steps of the detection target reconstruction method according to any one of claims 1 to 8 are implemented.

Images