3D article identification method and system, and storage medium

Abstract

The invention provides a 3D article identification method and system, and a medium. The 3D article identification method comprises the following steps: rendering an input 3D Gaussian instance to view angle images corresponding to a plurality of view angles; calculating a visibility score of the 3D Gaussian instance in each view angle image according to the opacity and transmissivity of each Gaussian point of the 3D Gaussian instance in each view angle image; determining M view angle images with the highest visibility score as key frame images; re-segmenting a projection area in the key frame image by using SAM to obtain a mask area of each key frame image; semantic information of the mask areas in the key frame images is extracted, and a voting mechanism is adopted to fuse the semantic information of the mask areas in the key frame images so as to determine semantic information of the 3D Gaussian instance; and endowing the semantic information of the 3D Gaussian instance to Gaussian points in the 3D Gaussian instance, and outputting a 3D instance identification result containing complete semantic information.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a 3D object recognition method, a 3D object recognition system, and a computer readable storage medium. BACKGROUND

[0002] In the existing technology for recognizing 3D objects, there are problems of insufficient recognition accuracy, long recognition time, and poor adaptability. In the existing technology, when processing multi-view images, it is often difficult to efficiently and accurately obtain accurate segmentation masks and semantic information of objects.

[0003] The existing technology has a backward image selection mechanism and is inefficient. When recognizing objects, the existing technology cannot automatically select appropriate images to accurately recognize objects, and is highly dependent on manual image data screening. This manual selection method not only requires a large amount of manpower and time cost, and has a huge workload, but also due to the subjectivity and limitations of manual judgment, selection bias is likely to occur, resulting in a significant reduction in the accuracy of the final recognition. For example, in the intelligent warehouse management scenario, the types of goods stored in the warehouse are diverse and the quantity is large. Manual image selection requires workers to view a large number of images one by one, which not only consumes time, but also is prone to omissions. Moreover, due to the subjectivity of manual image selection, different personnel may have different selection standards, which makes it difficult for the selected images to accurately represent the characteristics of the objects, thereby affecting the subsequent recognition accuracy.

[0004] In addition, the existing technology has defects in fusing multi-source semantic information, and the accuracy is poor. It is difficult to effectively integrate the semantic information extracted by each model, and it is difficult to fully utilize the advantages of different models. Each model has its own limitations when used alone. For example, although the target detection framework can quickly locate objects, it lacks depth and breadth in semantic understanding. The image semantic understanding model has weak accurate positioning ability for objects in complex backgrounds. Due to the lack of effective fusion strategy, the existing technology is difficult to accurately recognize the semantics of 3D objects in complex scenes, resulting in a significant reduction in the accuracy of the recognition result.

[0005] Due to the difficulty of the existing technology to adapt to complex and variable application scenarios, the existing technology has poor adaptability when dealing with complex and variable application scenarios. In actual applications, such as robots performing navigation tasks in complex indoor and outdoor environments, intelligent monitoring systems dealing with changes in lighting and object occlusion, etc., the existing technology performs unsatisfactorily. Due to the inability to effectively handle factors such as lighting changes, occlusions, and similar object interference, the existing technology is prone to misidentification or missed identification in these complex scenarios, greatly limiting the widespread application of the existing technology in robot vision navigation, intelligent security, virtual reality, and other fields.

[0006] In order to overcome the above-mentioned defects of the prior art, the technical field urgently needs a 3D object recognition technology which can effectively reduce the calculation amount and the recognition time of the object, improve the accuracy and robustness of 3D object recognition, and meet the 3D object recognition requirements in complex scenes. SUMMARY

[0007] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0008] In order to overcome the above-mentioned defects of the prior art, the present application provides a 3D object recognition method, a 3D object recognition system, and a computer readable storage medium, which can effectively reduce the calculation amount and the recognition time of the object, improve the accuracy and robustness of 3D object recognition, and meet the 3D object recognition requirements in complex scenes.

[0009] Specifically, the above-mentioned 3D object recognition method according to the first aspect of the present application comprises the steps of: rendering an input 3D Gaussian instance to a plurality of perspective images corresponding to a plurality of perspectives; calculating a visibility score of the 3D Gaussian instance in each of the perspective images according to the opacity and the transmittance of each Gaussian point of the 3D Gaussian instance in each of the perspective images; determining M perspective images with the highest visibility scores as key frame images; re-segmenting a projection region in each of the key frame images by using a SAM to obtain a mask region of each of the key frame images; extracting semantic information of the mask region in each of the key frame images and fusing the semantic information of the mask region in each of the key frame images by using a voting mechanism to determine semantic information of the 3D Gaussian instance; and assigning the semantic information of the 3D Gaussian instance to each Gaussian point in the 3D Gaussian instance, and outputting a 3D instance recognition result containing complete semantic information.

[0010] Preferably, in an embodiment of the present application, the step of calculating the visibility score of the 3D Gaussian instance in each of the perspective images according to the opacity and the transmittance of each Gaussian point of the 3D Gaussian instance in each of the perspective images comprises the steps of: determining a rendering weight of each Gaussian point according to the opacity and the transmittance of each Gaussian point of the 3D Gaussian instance in each of the perspective images; determining that the Gaussian point is a visible Gaussian point when the rendering weight of the Gaussian point is greater than a visibility threshold; and counting the number of the visible Gaussian points of the 3D Gaussian instance in each of the perspective images to determine the visibility score of the 3D Gaussian instance in each of the perspective images.

[0011] Preferably, in an embodiment of the present application, the step of calculating the visibility score of the 3D Gaussian instance in each of the view images according to the opacity and transmittance of each Gaussian point of the 3D Gaussian instance comprises: setting the opacity of the jth Gaussian point of the 3D Gaussian instance in the ith view image as , and the transmittance as , wherein the transmittance is calculated according to the following formula: , wherein is the opacity of the kth Gaussian point of the 3D Gaussian instance in the ith view image, and the rendering weight of the jth Gaussian point is determined according to the opacity and the transmittance of the jth Gaussian point of the 3D Gaussian instance in the ith view image: , and the rendering weight of the jth Gaussian point is determined according to the opacity and the transmittance of the jth Gaussian point of the 3D Gaussian instance in the ith view image: , wherein is the number of visible Gaussian points of the 3D Gaussian instance in the ith view image, and the total number of Gaussian points of the 3D Gaussian instance in the ith view image is

[0012] Preferably, in an embodiment of the present application, the step of re-segmenting the projection region in the key frame image using SAM to obtain the mask region of each key frame image comprises: re-segmenting the projection region in the key frame image using SAM; and post-processing and optimizing the segmentation result through morphological operation to obtain the mask region of each key frame image.

[0013] Preferably, in an embodiment of the present application, the step of extracting the semantic information of the mask region in the key frame image and fusing the semantic information of the mask region in each key frame image using a voting mechanism to determine the semantic information of the 3D Gaussian instance comprises: extracting the label information of YOLO and the feature information of CLIP from the mask region of each key frame image, respectively; determining the semantic information of the mask region in each key frame image based on the label information of YOLO and the feature information of CLIP corresponding to the mask region of each key frame image; and fusing the semantic information of the mask region in each key frame image according to a voting formula to determine the semantic information of the 3D Gaussian instance.

[0014] Preferably, in an embodiment of the present application, the voting formula is: wherein, is a confidence of semantic information of a mask region in the p-th extracted key frame image, is a candidate semantic, is a Kronecker function when , , otherwise .

[0015] Preferably, in an embodiment of the present application, the step of assigning the semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance to output the 3D instance recognition result containing complete semantic information comprises: calculating coordinates of each Gaussian point in the 3D Gaussian instance in the key frame image according to camera internal and external parameters; and assigning the semantic information of the 3D Gaussian instance corresponding to the mask region in the key frame image to the Gaussian points falling in the mask region in the key frame image.

[0016] Further, the above-mentioned 3D object recognition system according to the second aspect of the present application comprises a memory and a processor. The memory has computer instructions stored thereon. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the 3D object recognition method provided by any one of the above-mentioned embodiments.

[0017] Preferably, in an embodiment of the present application, the 3D object recognition system comprises: a 3D instance rendering module configured to render the input 3D Gaussian instance to a plurality of perspective images corresponding to different perspectives; a visibility score and key frame selection module configured to calculate a visibility score of the 3D Gaussian instance in each of the perspective images according to the opacity and transmittance of each Gaussian point of the 3D Gaussian instance in each of the perspective images, and determine M perspective images with the highest visibility scores as key frame images; a SAM re-segmentation module configured to re-segment the projection region in the key frame images using SAM to obtain mask regions of each of the key frame images; a semantic extraction and fusion module configured to extract semantic information of the mask regions in the key frame images and fuse the semantic information of the mask regions in each of the key frame images using a voting mechanism to determine the semantic information of the 3D Gaussian instance; and a semantic injection module configured to assign the semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance to output the 3D instance recognition result containing complete semantic information.

[0018] Further, the above-mentioned computer readable storage medium according to the third aspect of the present application has computer instructions stored thereon. The computer instructions are executed by a processor to implement the 3D object recognition method provided by any one of the above-mentioned embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0020] Figure 1 A flowchart of a 3D object recognition method according to some embodiments of the present invention is shown;

[0021] Figure 2 A schematic diagram of a 3D article recognition method according to some embodiments of the present invention is shown;

[0022] Figure 3 A schematic diagram of a 3D article recognition system according to some embodiments of the present invention is shown; and

[0023] Figure 4 A schematic diagram of a 3D article recognition system according to some embodiments of the present invention is shown.

[0024] Figure label:

[0025] 100: Methods for recognizing 3D objects;

[0026] S110~S150: Steps;

[0027] 300: 3D object recognition system;

[0028] 310: Memory;

[0029] 311: Computer-readable storage media; and

[0030] 320: Processor. Detailed Implementation

[0031] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be noted that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0032] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0033] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0034] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0035] As mentioned above, existing 3D object recognition technologies suffer from insufficient accuracy, long processing times, and poor adaptability. When processing multi-view images, existing technologies often struggle to efficiently and accurately acquire semantic information and precise segmentation masks for objects. Their image selection mechanisms are outdated and inefficient; they also have shortcomings in fusing multi-source semantic information, resulting in poor accuracy and an inability to fully leverage the strengths of different models or effectively integrate the semantic information extracted by each model. Furthermore, existing technologies are ill-suited to complex and ever-changing application scenarios, thus exhibiting poor adaptability in addressing such situations.

[0036] To overcome the aforementioned deficiencies in the existing technology, the present invention provides a 3D object recognition method, a 3D object recognition system, and a computer-readable storage medium, which can effectively reduce the computational load and recognition time of the object, improve the accuracy and robustness of 3D object recognition, and meet the 3D object recognition needs in complex scenarios.

[0037] In some non-limiting embodiments, the 3D article recognition method provided in the first aspect of the present invention can be implemented via the 3D article recognition system provided in the second aspect of the present invention. Specifically, the 3D article recognition system may be configured with a memory and a processor. The memory includes, but is not limited to, the computer-readable storage medium provided in the third aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and configured to execute the computer instructions stored in the memory to implement the 3D article recognition method provided in the first aspect of the present invention.

[0038] The working principle of the 3D object recognition system will be described below with reference to some embodiments of 3D object recognition methods. Those skilled in the art will understand that these embodiments of 3D object recognition methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or working methods of the 3D object recognition system. Similarly, the 3D object recognition system is also only one non-limiting implementation provided by the present invention, and does not limit the executing entities and execution order of the steps in these 3D object recognition methods.

[0039] Please refer to Figure 1 and Figure 2 , Figure 1 A flowchart of a 3D object recognition method according to some embodiments of the present invention is shown. Figure 2 A schematic diagram of a 3D article recognition method provided according to some embodiments of the present invention is shown.

[0040] like Figure 1 As shown, the 3D object recognition method 100 may include step S110: rendering the input 3D Gaussian instance to view images corresponding to multiple viewpoints.

[0041] 3D Gaussian Splatting (3DGS) is a real-time scene reconstruction and rendering technology based on explicit 3D Gaussian point cloud representation.

[0042] like Figure 2 As shown, the 3D object recognition system (hereinafter referred to as the "recognition system") can receive input 3D Gaussian instances, render the 3D Gaussian instances onto images corresponding to all viewpoints to determine the instance rendering image, thereby obtaining an image data set containing multi-viewpoint information. Here, all viewpoints refer to all possible angles around the object that can be observed when the user takes the picture.

[0043] The total number of multiple viewpoints can be N. After rendering, the recognition system can obtain an image set formed by viewpoint images from multiple viewpoints: This image set can provide a rich multi-view raw data foundation for subsequent keyframe image selection and object recognition steps, and the 3D Gaussian instances in the view images under multiple viewpoints in the formed image set are geometrically corresponding.

[0044] After that, as Figure 1 As shown, the 3D object recognition method 100 may include step S120: calculating the visibility score of the 3D Gaussian instance in each view image based on the opacity and transmittance of each Gaussian point of the 3D Gaussian instance in each view image.

[0045] The recognition system can use algorithms to calculate the visibility score of 3D Gaussian instances in images from different viewpoints.

[0046] In some embodiments, the recognition system may first determine the number of Gaussian points of 3D Gaussian instances visible in each viewpoint image under that viewpoint, and score the visibility of the viewpoint image based on the number of Gaussian points of 3D Gaussian instances visible under that viewpoint.

[0047] Specifically, the recognition system can determine the rendering weight of each Gaussian point based on the opacity and transmissivity of each Gaussian point in the 3D Gaussian instance in images from different viewpoints. Here, the transmissivity of a Gaussian point is used to represent the light transmission of that Gaussian point in the direction of the camera light, that is, to indicate whether there is occlusion in front of that Gaussian point.

[0048] When the rendering weight of a Gaussian point is greater than the visibility threshold, the Gaussian point is determined to be a visible Gaussian point. The rendering weight of a Gaussian point is used to indicate the degree to which the Gaussian point ultimately participates in the rendering of the corresponding pixel. When the weight value is 0, it means that the Gaussian point has no contribution to the pixel rendering; the higher the weight value, the greater the contribution of the Gaussian point to the pixel rendering.

[0049] In one example, the recognition system can set the opacity of the j-th Gaussian point of the 3D Gaussian instance in the i-th view image to be . Transmittance Among them, transmittance The calculation formula is:

[0050] ,

[0051] in, Let be the opacity of the k-th Gaussian point of the 3D Gaussian instance in the i-th view image.

[0052] Based on the opacity of the j-th Gaussian point of the 3D Gaussian instance in the i-th view image. and transmittance The recognition system can determine the rendering weight of the j-th Gaussian point. :

[0053] .

[0054] The recognition system can set a visibility threshold. When rendering weights Greater than or equal to the visibility threshold When the j-th Gaussian point is determined to be a visible Gaussian point, that is, the Gaussian point is considered to be a visible point.

[0055] Then, the recognition system can count the number of visible Gaussian points of 3D Gaussian instances in each viewpoint image to determine the visibility score of 3D Gaussian instances in each viewpoint image.

[0056] In the example above, the recognition system can set the total number of 3D Gaussian instances in the i-th viewpoint image as . Let the number of visible Gaussian points of a 3D Gaussian instance in the i-th viewpoint image be . Therefore, the visibility score of the 3D Gaussian instance from this perspective. It can be calculated using the following formula:

[0057] .

[0058] Please continue to refer to this. Figure 1 The 3D object recognition method 100 may include step S130: determining the M viewpoint images with the highest visibility scores as keyframe images.

[0059] After determining the visibility score of the 3D Gaussian instance in each viewpoint image, the recognition system can sort the viewpoint images according to the visibility score and identify the M viewpoint images with the highest visibility scores as keyframe images. The set of keyframe images formed by the M viewpoint images with the highest visibility scores can be used to represent image frames of the same object at different times / different shooting angles.

[0060] In the example above, the recognition system can use visibility scoring. After quantifying the visibility of 3D Gaussian instances from various viewpoints using the calculation formula, a visibility score is awarded based on the calculated results. The system sorts all the images from different perspectives and automatically selects the M images with the highest scores as keyframes.

[0061] Through the visibility filtering mechanism established in steps S120-S130, the recognition system can abandon the traditional manual image selection mode. It utilizes 3DGS technology to calculate the opacity and transmittance of Gaussian points, thereby comprehensively evaluating the visibility of 3D Gaussian instances from various viewpoints. By setting a visibility threshold to determine whether each Gaussian point is visible, a visibility score is determined, and keyframe images are automatically selected accordingly. In this way, the recognition system can accurately and efficiently select images that best represent the characteristics of 3D Gaussian instances, avoiding subjective biases from manual image selection, effectively eliminating redundant viewpoint images, significantly reducing the amount of data and computational complexity in subsequent processing, improving the scientific rigor and efficiency of keyframe image selection, and accelerating the subsequent recognition process.

[0062] Then, the recognition system can perform step S140: use SAM to re-segment the projection region in the keyframe image to obtain the mask region of each keyframe image.

[0063] SAM (Segment Anything Model) is an image segmentation model. The recognition system can take keyframe images as input and use SAM to re-segment the projection regions in the keyframe images to obtain the mask regions of each keyframe image.

[0064] In a preferred embodiment, after the recognition system re-segments the projection region in the keyframe image using SAM, it can further optimize the segmentation result through post-processing methods such as morphological operations to obtain an accurate and multi-view consistent instance 2D mask. That is, in the masks of different keyframe images, the same item is marked as the same 3D Gaussian instance by the recognition system, thereby clarifying which item the mask of different frames corresponds to, and inferring or stitching out the true 3D outline of the item based on these masks.

[0065] In one example, the set of keyframe images determined in step S130 can be After the keyframe images are re-segmented and post-processed by SAM, the corresponding mask set is obtained. The masked regions of each keyframe image that have been determined It can provide reliable data for accurate identification of object outlines in the future.

[0066] Please continue to refer to this. Figure 1 The 3D object recognition method 100 may include step S150: extracting the semantic information of the mask region in the keyframe image and using a voting mechanism to fuse the semantic information of the mask region in each keyframe image to determine the semantic information of the 3D Gaussian instance.

[0067] In some embodiments, the recognition system can utilize the YOLO (You Only Look Once) model and the CLIP (Contrastive Language-Image Pretraining) model to extract semantic information of masked regions in keyframe images. The YOLO model is a deep learning-based real-time object detection framework that transforms the object detection task into a regression problem, directly outputting the bounding box coordinates, confidence scores, and class probabilities of the object detection task using a single-pass forward propagation convolutional neural network (CNN). The CLIP model is a technique that enables models to understand the semantic relationships between images and text through large-scale pre-training.

[0068] The recognition system can extract YOLO label information and CLIP feature information from the masked regions of keyframe images as semantic information of the masked regions. The system combines YOLO's fast classification capabilities with CLIP's multimodal feature extraction advantages to extract different types of semantic information from the masked regions of keyframe images. This semantic information includes semantic label information and the corresponding feature vector information.

[0069] Subsequently, the recognition system can use a voting mechanism to fuse semantic information from multiple perspectives and sources, thereby determining the final 3D Gaussian instance-level semantic information.

[0070] Specifically, the recognition system first extracts YOLO label information and CLIP feature information from the masked regions of each keyframe image. Then, based on the YOLO label information and CLIP feature information corresponding to the masked regions of each keyframe image, the semantic information of the masked regions in each keyframe image is determined. Finally, the semantic information of the masked regions in each keyframe image is fused according to a voting formula to determine the semantic information of the 3D Gaussian instance.

[0071] In one example, the recognition system can determine the semantic information of the masked region in the p-th keyframe image as follows: The semantic set obtained from M keyframe images is Semantic information can be used to "label" the masked area to identify what item is inside. Furthermore, the recognition system can perform preliminary detection using the YOLO model and semantic verification using the CLIP model to determine the category label and obtain semantic information.

[0072] The recognition system can determine semantic information The corresponding feature vector is a representation of semantic information. The quantified representation of the visual features of the corresponding object is typically a vector with hundreds or thousands of dimensions. Semantic information The corresponding feature vectors can be obtained through the image feature encoder of the CLIP model.

[0073] Then, the recognition system can determine the semantic label of the final 3D Gaussian instance using a voting formula. :

[0074] ,

[0075] in, Let be the confidence level of the semantic information of the masked region in the extracted p-th keyframe image. To indicate whether the tag to be determined is the final semantic tag The candidate semantics correspond to the semantic set mentioned above. ,when Let Kronecker function be used when hour, ,otherwise The purpose of the voting formula is to select from all possible candidate semantic sets. In the process, semantic labels that can ultimately represent 3D Gaussian instances are selected. The voting formula calculates each candidate semantic Based on the corresponding voting scores, identify the candidate semantics with the highest score. This was determined as the final semantic tag. .

[0076] Here, the confidence level of the semantic information of the masked region in the p-th keyframe image. It measures semantic information Accuracy is a quantitative indicator, typically ranging from 0 to 1, with values ​​closer to 1 indicating greater semantic information. The more reliable. The confidence level of the semantic information of the masked region in the p-th keyframe image. The detection probability of the YOLO model and the image-text similarity of the CLIP model can be combined to calculate and determine the result, thus avoiding the bias of a single model.

[0077] Through the voting mechanism described above, the recognition system can comprehensively consider the confidence level of the semantic information of each key frame, fuse the semantic information extracted from the masked regions in each key frame image, and thus fuse semantic information from multiple perspectives and sources to finally determine the accurate semantics of the 3D Gaussian instance.

[0078] Furthermore, through step S140, the recognition system can also integrate the advantages of different models to significantly improve the accuracy and reliability of semantic recognition, enabling the recognition system to accurately identify the complete semantic information of 3D objects even in complex scenarios.

[0079] Finally, the recognition system can perform step S150: assign the semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance, and output the 3D instance recognition result containing complete semantic information.

[0080] For example, in the example above, the recognition system can identify semantic tags. and semantic tags The corresponding feature vectors are assigned to Gaussian points in the 3D Gaussian instance to complete instance-level semantic injection. Thus, the recognition system can accurately identify 3D objects and output 3D instance recognition results containing complete semantic information.

[0081] Specifically, each 3D Gaussian instance of a 3D object includes multiple Gaussian points, which in turn correspond to a semantically labeled mask region in the 2D image. The semantic labels and feature vectors of this region are "mapped" to the Gaussian points corresponding to the 3D Gaussian instance to complete semantic injection.

[0082] In some embodiments, the recognition system can first calculate the 2D projected pixel coordinates of Gaussian points in each 3D Gaussian instance in each keyframe based on the camera's intrinsic and extrinsic parameters, then filter out Gaussian points whose projected pixel coordinates fall within the keyframe mask area, and assign the semantic label and feature vector of the 3D Gaussian instance corresponding to this mask area as the attribute of the corresponding Gaussian point to the Gaussian point falling within the keyframe mask area.

[0083] In summary, the 3D object recognition method provided by this invention not only solves the problems of manual image selection bias and insufficient semantic fusion, but also enhances the adaptability of the recognition system to complex environments, such as changes in lighting and partial occlusion, significantly improving the accuracy and robustness of 3D object recognition and broadening the scope of its application. The automation and efficiency of the 3D object recognition method provided by this invention are significantly improved. Based on visibility scoring, keyframe images in each viewpoint are determined, completely eliminating the tediousness and inefficiency of manual image selection, greatly reducing labor costs and time consumption. Simultaneously, it reduces redundant data processing, significantly shortening the overall time for 3D object recognition, with particularly obvious advantages in scenarios involving massive data processing. The recognition accuracy of the 3D object recognition method provided by this invention is greatly improved. This invention obtains accurate object masks through a SAM model and combines a multi-source semantic deep fusion strategy, effectively avoiding recognition errors caused by manual image selection bias and insufficient semantic fusion in traditional methods. High-precision recognition can be achieved for 3D objects in both simple scenes and complex backgrounds. Furthermore, the interactive experience and adaptability of the 3D object recognition method provided by this invention are enhanced. In complex and ever-changing scenarios, this invention can still maintain stable and reliable recognition performance, and can give full play to the advantages of different models, enhancing adaptability to diverse objects and complex environments.

[0084] The 3D object recognition system provided by this invention can, as Figure 3 As shown, the 3D object recognition system 300 is configured with a memory 310 and a processor 320. The memory 310 includes, but is not limited to, the computer-readable storage medium 311 provided by the present invention, which stores computer instructions thereon. The processor 320 is connected to the memory 310 and is configured to execute the computer instructions stored in the memory 310 to implement the 3D object recognition method provided by the present invention.

[0085] In other embodiments, the 3D object recognition system provided by the present invention can also be as follows: Figure 4As shown, a 3D object recognition system may include multiple program modules. These program modules may be stored in memory. Program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data; each or some combination of these examples may include an implementation of a network environment. Program modules are typically used to perform the functions and / or methods described in the embodiments of this invention.

[0086] like Figure 4 As shown, a 3D object recognition system may include a 3D instance rendering module, a visibility scoring and keyframe selection module, a SAM resegmentation module, a semantic extraction and fusion module, and a semantic injection module.

[0087] The 3D instance rendering module can render the input 3D Gaussian instance into view images corresponding to multiple viewpoints. The rendered view images are then input into the visibility scoring and keyframe selection module.

[0088] The visibility scoring and keyframe selection module can calculate the visibility score of the 3D Gaussian instance in each viewpoint image based on the opacity and transmittance of each Gaussian point in the 3D Gaussian instance, and determine the M viewpoint images with the highest visibility scores as keyframe images.

[0089] The visibility scoring and keyframe selection module inputs the determined keyframe images into the SAM re-segmentation module. The SAM re-segmentation module uses SAM to re-segment the projection regions in the keyframe images to obtain the 2D mask regions of each keyframe image.

[0090] The SAM resegmentation module inputs the 2D mask regions of each keyframe image to the semantic extraction and fusion module. The semantic extraction and fusion module can extract the semantic information of the mask regions in the keyframe images and use a voting mechanism to fuse the semantic information of the mask regions in each keyframe image, and determine the fused semantic information as the semantic information of the 3D Gaussian instance.

[0091] The semantic extraction and fusion module inputs the fused semantic information into the semantic injection module. The semantic injection module can assign the semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance and output the 3D instance recognition result containing complete semantic information.

[0092] exist Figure 4 In the illustrated embodiment, through the close cooperation and coordinated operation of the five modules, the system can automatically select suitable images, obtain accurate object masks, improve the reliability of semantic recognition, effectively reduce the amount of computation and recognition time, improve the accuracy and robustness of 3D object recognition, meet the needs of 3D object recognition in complex scenarios, and jointly achieve efficient and accurate 3D object recognition. At the same time, the efficient operation of each module can also bring a smoother recognition interaction experience.

[0093] In summary, the 3D object recognition method and 3D object recognition system provided by the invention can effectively reduce the computational load and recognition time of objects, improve the accuracy and robustness of 3D object recognition, and meet the 3D object recognition needs in complex scenarios.

[0094] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0095] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and skills. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0096] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0097] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0098] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0099] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0100] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing 3D objects, characterized in that, Including the following steps: Render the input 3D Gaussian instance to view images corresponding to multiple viewpoints; Based on the opacity and transmittance of each Gaussian point of the 3D Gaussian instance in each of the aforementioned viewpoint images, calculate the visibility score of the 3D Gaussian instance in each of the aforementioned viewpoint images. The M viewpoint images with the highest visibility scores are identified as keyframe images; The projection region in the keyframe image is re-segmented using SAM to obtain the mask region of each keyframe image. Semantic information of the masked regions in the keyframe images is extracted and fused using a voting mechanism to determine the semantic information of the 3D Gaussian instance; and The semantic information of the 3D Gaussian instance is assigned to the Gaussian points in the 3D Gaussian instance, and the 3D instance recognition result containing complete semantic information is output.

2. The 3D object recognition method as described in claim 1, characterized in that, The step of calculating the visibility score of the 3D Gaussian instance in each viewpoint image based on the opacity and transmittance of each Gaussian point of the 3D Gaussian instance in each viewpoint image includes: The rendering weight of each Gaussian point is determined based on the opacity and transmissivity of each Gaussian point in the 3D Gaussian instance in each viewpoint image. When the rendering weight of the Gaussian point is greater than the visibility threshold, the Gaussian point is determined to be a visible Gaussian point; and The number of visible Gaussian points of 3D Gaussian instances in each viewpoint image is counted to determine the visibility score of 3D Gaussian instances in each viewpoint image.

3. The 3D object recognition method as described in claim 2, characterized in that, The step of calculating the visibility score of the 3D Gaussian instance in each viewpoint image based on the opacity and transmittance of each Gaussian point of the 3D Gaussian instance includes: Let the opacity of the j-th Gaussian point of the 3D Gaussian instance in the i-th view image be . Transmittance , wherein the transmittance The calculation formula is: ， in, Let k be the opacity of the k-th Gaussian point of the 3D Gaussian instance in the i-th viewpoint image; Based on the opacity of the j-th Gaussian point of the 3D Gaussian instance in the i-th view image. and transmittance Determine the rendering weight of the j-th Gaussian point. : ， When rendering weight Greater than or equal to the visibility threshold When the j-th Gaussian point is determined to be a visible Gaussian point; The visibility score of a 3D Gaussian instance in the i-th viewpoint image is determined by counting the number of visible Gaussian points of the instance. : ， in, Let be the number of visible Gaussian points of the 3D Gaussian instance in the i-th viewpoint image. The total number of Gaussian points in the 3D Gaussian instance in the i-th viewpoint image.

4. The 3D object recognition method as described in claim 1, characterized in that, The step of re-segmenting the projection region in the keyframe image using SAM to obtain the mask region of each keyframe image includes: The projection region in the keyframe image is re-segmented using SAM; and The segmentation results are post-processed and optimized using morphological operations to obtain the mask regions of each keyframe image.

5. The 3D object recognition method as described in claim 1, characterized in that, The step of extracting the semantic information of the masked regions in the keyframe images and fusing the semantic information of the masked regions in each keyframe image using a voting mechanism to determine the semantic information of the 3D Gaussian instance includes: YOLO label information and CLIP feature information are extracted from the mask region of each keyframe image respectively; Based on the YOLO label information and CLIP feature information corresponding to the masked regions of each keyframe image, the semantic information of the masked regions in each keyframe image is determined; and The semantic information of the masked regions in each keyframe image is fused according to the voting formula to determine the semantic information of the 3D Gaussian instance.

6. The 3D object recognition method as described in claim 5, characterized in that, The voting formula is as follows: ， in, Let be the confidence level of the semantic information of the masked region in the extracted p-th keyframe image. Candidate semantics, Let Kronecker function be used when hour, ,otherwise .

7. The 3D object recognition method as described in claim 1, characterized in that, The step of assigning semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance and outputting a 3D instance recognition result containing complete semantic information includes: Based on the camera's intrinsic and extrinsic parameters, calculate the coordinates of each Gaussian point in the 3D Gaussian instance within the keyframe image; and The semantic information of the 3D Gaussian instance corresponding to the mask region in the keyframe image is assigned to the Gaussian point falling in the mask region in the keyframe image.

8. A 3D object recognition system, characterized in that, include: Memory, on which computer instructions are stored; as well as A processor, connected to the memory, and configured to execute computer instructions stored on the memory to implement the 3D article recognition method as described in any one of claims 1 to 7.

9. The 3D article recognition system as described in claim 8, characterized in that, include: The 3D instance rendering module is configured to render the input 3D Gaussian instance to view images corresponding to multiple viewpoints; The visibility scoring and keyframe selection module is configured to calculate the visibility score of the 3D Gaussian instance in each viewpoint image based on the opacity and transmittance of each Gaussian point of the 3D Gaussian instance in each viewpoint image, and determine the M viewpoint images with the highest visibility scores as keyframe images. The SAM resegmentation module is configured to resegment the projection regions in the keyframe images using SAM to obtain the mask regions of each keyframe image. The semantic extraction and fusion module is configured to extract the semantic information of the masked regions in the keyframe images and fuse the semantic information of the masked regions in each keyframe image using a voting mechanism to determine the semantic information of the 3D Gaussian instance. as well as The semantic injection module is configured to assign semantic information of the 3D Gaussian instance to the Gaussian points in the 3D Gaussian instance and output a 3D instance recognition result containing complete semantic information.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed by the processor, the method for recognizing 3D articles as described in any one of claims 1 to 7 is implemented.

Images