Object posture recognition model, recognition method, device and system
By combining image segmentation, mask mapping, feature extraction and coding matching modules in the object position recognition model, and combining PnP algorithm, the problems of high training cost and high inference cost in the existing technology are solved, and the multi-object model is achieved Efficient recognition and general recognition of individual object positions.
Patent Information
- Application Number
- CN202411001232.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The existing object position estimation algorithm based on deep neural networks is expensive to train and inferencing in multi-object model scenarios, lacks universality, and cannot effectively identify the position poses of different objects.
An object position recognition model is proposed, including image segmentation module, mask mapping module, feature extraction module, related calculation module and encoding matching module. Through these modules, the feature extraction and encoding matching of object images are realized, and the objects are obtained by combining PnP algorithm. Position, can train multiple object models at the same time and do not change network weight parameters during the inference stage.
It reduces the training and inference costs, improves the learning efficiency and practicality of the network, and realizes universal recognition of multiple object positions.
Smart Images

Figure CN118537409B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to an object posture recognition model, recognition method, device and system. Background Art
[0002] The main principle of the existing image-based pose estimation algorithm is to segment the object image through a deep neural network, match the pixel 2D points in the object image with the 3D vertices on the object model, and then use the PnP algorithm to solve the pose of the object in the current image.
[0003] However, when using deep neural network training, only a single object is trained each time, and a set of network parameters corresponds to only one object model. When the data set contains multiple object models, it is necessary to learn them multiple times, which increases the training cost. At the same time, different network parameters need to be replaced to reason about different object models, which increases the reasoning cost and is not conducive to the actual application of the project. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an object posture recognition model, recognition method, device and system to improve the versatility of recognition for different object postures.
[0005] In a first aspect, the present application provides an object posture recognition model, which adopts the following technical solution:
[0006] An object posture recognition model, the model comprising: an image segmentation module, a mask mapping module, a feature extraction module, a correlation calculation module and a coding matching module;
[0007] The image segmentation module is used to segment the object image to obtain the mask image of the object and the region of interest of the object;
[0008] The mask mapping module is used to map the mask image of the object to a continuous space to generate a query vector;
[0009] The feature extraction module is used to extract a feature map of an object in an object image based on a region of interest of the object;
[0010] The correlation calculation module is used to perform correlation calculation on the query vector and the feature map of the object to obtain the encoding of the pixel coordinates of the object;
[0011] The code matching module is used to match the code of the pixel point coordinates of the object with the 3D code of the vertices of the object model to obtain the corresponding relationship between the 2D pixel points and the 3D points of the object.
[0012] Furthermore, in one embodiment, the model further comprises:
[0013] The posture obtaining module is used to obtain the posture of the object in combination with the corresponding relationship.
[0014] Further, in one embodiment, the posture obtaining module includes:
[0015] A feature extraction submodule, for extracting 2D features of an object based on a feature map of the object;
[0016] The information obtaining submodule is used to obtain feature information based on the 2D feature: the feature information includes: weight distribution of 2D feature points, 2D point coordinates and 3D vertex coordinates.
[0017] The PnP submodule is used to obtain the object posture by combining the corresponding relationship and the feature information.
[0018] In a second aspect, an embodiment of the present application provides a method for object posture recognition, the method comprising the following steps:
[0019] Acquire an object image;
[0020] The object image is used as the input of the object posture recognition model described above, and the corresponding relationship between the 2D pixel points and the 3D points of the object is obtained as output;
[0021] Based on the corresponding relationship, the position and posture of the object is obtained.
[0022] In a third aspect, an embodiment of the present application provides a method for object posture recognition, the method comprising the following steps:
[0023] Acquire an object image;
[0024] The object image is used as the input of the object posture recognition model described above, and the object posture is obtained as the output.
[0025] In a fourth aspect, an embodiment of the present application provides a posture recognition device, the device comprising:
[0026] An image acquisition module, used for acquiring an image of an object;
[0027] A relationship building module, used to use the object image as input of the object posture recognition model described above, and output the corresponding relationship between the 2D pixel points and 3D points of the object;
[0028] The posture obtaining module is used to obtain the posture of the object based on the corresponding relationship.
[0029] In a fifth aspect, an embodiment of the present application provides a posture recognition device, the device comprising:
[0030] An image acquisition module, used for acquiring an image of an object;
[0031] The posture obtaining module is used to take the object image as the input of the object posture recognition model mentioned above, and output the object posture.
[0032] In a sixth aspect, an embodiment of the present application provides a posture recognition system, the system comprising: an image sensor and a controller;
[0033] The image sensor is communicatively connected with the controller;
[0034] The controller is used to implement the object posture recognition method described above.
[0035] In a seventh aspect, an embodiment of the present application provides a controller comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the object posture recognition method described in any one of the above items are implemented.
[0036] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the object posture recognition method described in any one of the above items are implemented.
[0037] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0038] The embodiment of the present application can simultaneously train to recognize the postures of multiple objects based on the same neural network model, thereby reducing the training cost and improving the learning efficiency of the network; in addition, there is no need to change the network weight parameters during the network reasoning stage, thereby reducing the network prediction cost and improving the network practicality. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0040] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0041] Figure 2 is a structural block diagram of an embodiment of the object posture recognition model of the present application;
[0042] Figure 3 A structural block diagram of another embodiment of the object posture recognition model of the present application;
[0043] Figure 4It is a flowchart of an embodiment of the object posture recognition method of the present application;
[0044] Figure 5 is a flowchart of another embodiment of the object posture recognition method of the present application;
[0045] Figure 6 It is a schematic diagram of the framework structure of an embodiment of the posture recognition device of the present application;
[0046] Figure 7 It is a schematic diagram of the framework structure of an embodiment of the posture recognition device of the present application;
[0047] Figure 8 It is a structural diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION
[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0049] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0050] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0051] like Figure 1 As shown, Figure 1 is an exemplary system architecture diagram to which the present application can be applied.
[0052] An embodiment of the present application provides an image capturing system 100 , which includes an image sensor 110 and a controller 120 .
[0053] The image sensor 120 is used to capture an image of an object.
[0054] It should be noted that the image sensor can be a monocular or multi-camera, a video camera, or a mobile phone, tablet computer, laptop computer, desktop computer or other device including a camera or a video camera.
[0055] The controller 120 is connected to the image sensor 110 in a wired or wireless communication manner.
[0056] For the limitation of the controller, please refer to the limitation of the object posture recognition method in the following embodiments.
[0057] Specifically, the controller 120 may generate program instructions according to a pre-fixed program in combination with data signals collected by the external image sensor 110 or the like.
[0058] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.
[0059] The controller provided in the embodiments of the present application may be, but is not limited to: a computer terminal (Personal Computer, PC); an industrial control computer terminal (Industrial Personal Computer, IPC); a mobile terminal; a server; a system including a terminal and a server, and implemented through the interaction between the terminal and the server; a programmable logic controller (Programmable Logic Controller, PLC); a field programmable gate array (Field-Programmable Gate Array, FPGA); a digital signal processor (Digital Signal Processer, DSP) or a microcontroller unit (Microcontroller unit, MCU) and the like. Specifically, the controller may be, for example, Figure 8 Computer equipment shown.
[0060] It should be noted that the controller described in the embodiments of the present application can be provided separately or partially integrated in devices such as image sensors, and both fall within the scope of protection of the present application.
[0061] Embodiment 1
[0062] like Figure 2 As shown, Figure 2 It is a structural block diagram of an embodiment of the object posture recognition model of the present application.
[0063] In one embodiment, the embodiment of the present application provides an object posture recognition model, which includes: an image segmentation module 210, a mask mapping module 220, a feature extraction module 230, a correlation calculation module 240 and a coding matching module 250.
[0064] The image segmentation module 210 is used to segment the object image to obtain the mask image of the object and the region of interest of the object.
[0065] The above-mentioned image segmentation module can be but is not limited to: FCN, Unet, Unet++, RefineNet or Yolo and other various network structures with similar functions that exist now or will be developed in the future.
[0066] In one embodiment, the object image may be used as an input of an image segmentation module, a region of interest ROI of the object may be output based on an intermediate layer of the image segmentation module, and a mask of the object may be obtained based on a final output of the image segmentation module.
[0067] In one embodiment, the object posture recognition model may further include: a preprocessing module.
[0068] The preprocessing module is used to preprocess the region of interest so as to input the preprocessed region of interest into the feature extraction module described in the following embodiments.
[0069] In one embodiment, the above preprocessing may include but is not limited to: clipping and / or pooling preprocessing, and a fixed-size region of interest may be obtained by clipping and pooling the region of interest. In addition, any other preprocessing method may be used as needed, which all fall within the scope of protection of this application.
[0070] The mask mapping module 220 is used to map the mask image of the object to a continuous space to generate a query vector.
[0071] The mask mapping module may be, but is not limited to, an Embedding layer and a fully connected layer. The mask image of the object is mapped to a continuous space through a mapping network such as an Embedding layer to obtain a query vector query.
[0072] In the embodiment of the present application, the mask mapping module can map high-dimensional data to low-dimensional space, for example, mapping the mask image to continuous space, and using a low-dimensional query vector to represent the high-dimensional data. The purpose of this is to use appropriate data to describe the spatial information of the points on the mask in the object model. This query vector is not just a string of numbers, usually each point in the mask corresponds to a query vector.
[0073] The feature extraction module 230 is used to extract a feature map of an object in the object image based on the region of interest of the object.
[0074] The feature extraction module may be, but is not limited to, ResNet, VGGNet, InceptionNet or other network structures with feature extraction functions that exist now or will be developed in the future.
[0075] The correlation calculation module 240 is used to perform correlation calculation on the query vector and the feature map of the object to obtain the encoding of the pixel coordinates of the object.
[0076] Specifically, the above-mentioned correlation calculation module can be, but is not limited to: Cross-Attention, MultiHeatAttention network structure, to calculate the correlation between the two inputs. In addition, it can also be calculated directly through a two-way attention algorithm to calculate the correlation between the query vector and the feature map of the object's region of interest. In the embodiment of the present application, as long as it is a method or network model that can calculate the correlation between the two inputs, it belongs to the scope of protection of this application.
[0077] Exemplarily, based on the above-mentioned correlation calculation module 240, binary codes of pixel point coordinates of various object models in the image can be generated to obtain a 2D point coding table, [(01100111, (u, i)), (01100110, (u,i)), ...]. Among them, the 2D point coordinates corresponding to the code (01100111) are (u, i), which are used to record the relationship between the pixel point coordinates of the object model in the image and the binary code.
[0078] The code matching module 250 is used to match the code of the pixel point coordinates of the object with the 3D code of the vertices of the object model to obtain the correspondence between the 2D pixel points and the 3D points of the object.
[0079] In one embodiment, the controller can classify and encode the coordinate points of the 3D model in advance through K-means clustering, such as 01100111, to generate a 3D point coding table in the form of binary code and corresponding 3D point coordinates, [(01100111, (x, y, z)), (01100110, (x, y, z)), ...], and store the 3D point coding table according to a preset address.
[0080] Based on the 2D point encoding table and the 3D point encoding table generated by the correlation calculation module 240 described in the above embodiment, the corresponding relationship between the 2D and 3D points can be constructed by looking up the table.
[0081] In an embodiment of the present application, if the object image includes multiple objects at the same time, the query vector obtained is for multiple objects and is classified. For example, if there are 3 objects, there are 3 query vectors. The feature map is not classified. It is all the regions of interest in the entire image. Finally, the feature map corresponding to the corresponding number of objects will be extracted. For example, only the feature map of 3 objects is retained. It is necessary to make a one-to-one correspondence between the 3 query vectors and the feature map. Through the correlation calculation method, the query vector with the highest correlation with the feature map, that is, the corresponding query vector, can be obtained.
[0082] The embodiments of the present application can simultaneously input object models of different categories during the training process, identify the object category through the image segmentation module, and after the image segmentation module, estimate the posture of the corresponding object according to the different categories of objects. Therefore, based on the same neural network model, training can be performed simultaneously to identify the postures of multiple objects, thereby reducing the training cost and improving the learning efficiency of the network. In addition, the recognition of the postures of different types of objects is achieved based on the same object posture recognition model, so there is no need to change the network weight parameters, thereby reducing the network prediction cost and improving the network practicality.
[0083] Embodiment 2
[0084] like Figure 3 As shown, Figure 3 It is a structural block diagram of another embodiment of the object posture recognition model of the present application.
[0085] In one embodiment, the object posture recognition model of the embodiment of the present application may include, in addition to: an image segmentation module 210, a mask mapping module 220, a feature extraction module 230, a correlation calculation module 240 and a coding matching module 250, also include: a posture acquisition module 260.
[0086] The posture obtaining module 260 is used to obtain the posture of the object based on the corresponding relationship.
[0087] Further, in one embodiment, the posture obtaining module 260 may include:
[0088] The feature extraction submodule 261 is used to extract the 2D features of the object based on the feature map of the object.
[0089] The information obtaining submodule 262 is used to obtain feature information based on 2D features: wherein the feature information includes: weight distribution of 2D feature points, 2D point coordinates and 3D vertex coordinates.
[0090] The posture obtaining submodule 263 is used to obtain the posture of the object by combining the corresponding relationship and the feature information.
[0091] Specifically, the above-mentioned posture obtaining submodule can adopt, but is not limited to: a PnP network structure layer.
[0092] Among them, the PnP algorithm is an algorithm for estimating the camera pose. Based on the three-dimensional point coordinates (x, y, z) in the above world coordinate system and the corresponding two-point pixel coordinates (i, j), the PnP algorithm can be used to obtain the object pose in the camera coordinate system.
[0093] Exemplarily, the input of the posture acquisition submodule is and 2d-3d corresponding points, where Represents 2d point coordinates; Represents 3d point coordinates; Represents 2d weight. The network output is the object's rotation pose quaternion H and the object's unique coordinate t.
[0094] In the embodiment of the present application, by considering the PnP algorithm for obtaining the optimal posture as the probability distribution of the posture, the network can perform back-propagation calculations.
[0095] The object posture recognition model of the embodiment of the present application can realize end-to-end object posture recognition by adopting a posture acquisition module; the optimal posture is regarded as the probability distribution of the posture, so that the network can perform back-propagation calculations. The entire network forms an end-to-end network structure, which can directly learn from the actual posture of the object.
[0096] It should be noted that in addition to the various sub-modules included in the posture acquisition module described in the above embodiment, the posture acquisition module can also adopt various existing or future developed modules and methods as needed. For example, the corresponding relationship can be used as the input of a pre-trained posture recognition network, and the object posture can be directly output. As long as the object posture can be obtained by combining the corresponding relationship, it falls within the scope of protection of this application.
[0097] Based on the image capture system and object posture recognition model of the above embodiment, the present application embodiment provides a method for training an object posture recognition model, which is generally executed by the controller 120. The above object posture recognition model training method may include the following method steps:
[0098] Step 510 obtains the current training sample.
[0099] In one embodiment, the controller obtains one of the current object image training samples from a training sample set consisting of multiple object images from a memory or a server based on a preset method according to a preset address.
[0100] Exemplarily, taking the training samples corresponding to the object posture recognition model in Example 1 as an example, the labels corresponding to each object image training sample include: label encoding of 2D pixel point coordinates; taking the training samples corresponding to the object posture recognition model in Example 2 as an example, the labels corresponding to each object image training sample include: label encoding of 2D pixel point coordinates, and label object posture.
[0101] It should be noted that the training samples only need to include: the label pose of the object, the label encoding of the 2D pixel point coordinates. Other mask images, regions of interest, query vectors, 2D features, feature information (including: 2D point coordinates and corresponding 3D point coordinates, weights) are all intermediate products of the model, and there is no need to calculate the true value of the label in advance. In addition, based on the above embodiment, the correspondence between 2D and 3D points is obtained by table lookup, so there is no need to obtain the true value of the label of the corresponding relationship in advance.
[0102] In one embodiment, for an object image, an object image training sample may be generated based on photographing by an image sensor or using a 3D model rendering of the object.
[0103] In one embodiment, before step 510, the tag object pose may be generated by the following method.
[0104] For real training sample object images, the label object pose can be obtained based on manual annotation. Based on the label object pose of the training sample-object image obtained by rendering, the object pose can be generated first, and then the 3D model can be used to render the training sample according to the object pose.
[0105] In one embodiment, before obtaining the current training sample in step 510, the label encoding of the 2D pixel point coordinates can be generated by the following method steps:
[0106] The 3D model of the object can be projected onto the 2D image with a known posture, the corresponding 2D point coordinates can be obtained, and then the binary code of the point coordinates can be obtained based on preset rules.
[0107] Step 520 iteratively trains the object pose recognition model based on the current training sample.
[0108] Specifically, various methods that exist now or will be developed in the future can be used to train the above model, such as supervised learning or semi-supervised learning.
[0109] Exemplarily, taking supervised learning as an example: a current object image training sample is used as the input of the object pose recognition model described in Example 2, and the binary code of the predicted pixel coordinates and the predicted object pose are output respectively; the difference between the binary code of the predicted pixel coordinates and the binary code of the label pixel coordinates is obtained (for example: cross entropy calculation is performed), as well as the pose difference between the predicted object pose and the label object pose (for example: the commonly used regression loss function can be used for calculation). The outputs of other intermediate layers of the model have no real values, so there is no need to calculate the difference. The parameters of the pose recognition model are iteratively optimized based on the code difference and the pose difference until the preset conditions are met (for example: each difference is simultaneously less than the preset threshold, or the iteration reaches a preset number of times).
[0110] The image capturing system according to the above embodiment and Figure 2 The posture recognition model shown, the embodiment of the present application provides an object posture recognition method, which is generally executed by the controller 120.
[0111] like Figure 4 As shown, Figure 4 : is a flow chart of an embodiment of the object posture recognition method of the present application; the object posture recognition method may include the following method steps:
[0112] Step 310 acquires an object image.
[0113] In one embodiment, the controller may obtain an object image acquired by an image sensor or an object image preprocessed by some method from a memory or a server according to a preset address.
[0114] In step 320, the object image is used as the input of the object posture recognition model described in the first embodiment, and the corresponding relationship between the 2D pixel points and the 3D points of the object is obtained as output.
[0115] Step 330 obtains the object's position and posture based on the corresponding relationship.
[0116] The embodiments of the present application can simultaneously input object models of different categories during the training process, identify the object category through the image segmentation module, and after the image segmentation module, estimate the posture of the corresponding object according to the different categories of objects. Therefore, based on the same neural network model, training can be performed simultaneously to identify the postures of multiple objects, thereby reducing the training cost and improving the learning efficiency of the network. In addition, the recognition of the postures of different types of objects is achieved based on the same object posture recognition model, so there is no need to change the network weight parameters, thereby reducing the network prediction cost and improving the network practicality.
[0117] The image capturing system according to the above embodiment and Figure 3The posture recognition model shown, the embodiment of the present application provides an object posture recognition method, which is generally executed by the controller 120.
[0118] like Figure 5 As shown, Figure 5 : is a flow chart of an embodiment of the object posture recognition method of the present application; the object posture recognition method may include the following method steps:
[0119] Step 410 acquires an object image.
[0120] In one embodiment, the controller may obtain an object image acquired by an image sensor or an object image preprocessed by some method from a memory or a server according to a preset address.
[0121] In step 420, the object image is used as input of the object posture recognition model described in the second embodiment, and the object posture is output.
[0122] The object posture recognition model of the embodiment of the present application can achieve end-to-end object posture recognition by adopting the object posture recognition model described in Example 2; the optimal posture is regarded as the probability distribution of the posture, so that the network can perform back-propagation calculations.
[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0124] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0125] Further references Figure 6 , as a response to the above Figure 4The present application provides an embodiment of an object posture recognition device, and the device embodiment is similar to Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied in a controller.
[0126] like Figure 6 As shown, the object posture recognition device 300 described in the embodiment of the present application includes:
[0127] An image acquisition module 310 is used to acquire an image of an object;
[0128] A relationship building module 320, used to use the object image as an input of the object posture recognition model described in the first embodiment, and output the corresponding relationship between the 2D pixel points and the 3D points of the object;
[0129] The first obtaining module 330 is used to obtain the object posture based on the corresponding relationship.
[0130] Further references Figure 7 , as a response to the above Figure 5 The present application provides an embodiment of an object posture recognition device, and the device embodiment is similar to Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied in a controller.
[0131] like Figure 7 As shown, the object posture recognition device 400 of this embodiment includes:
[0132] An image acquisition module 410 is used to acquire an image of an object;
[0133] The second obtaining module 420 is used to use the object image as the input of the object posture recognition model described in the second embodiment, and output the object posture.
[0134] Furthermore, the present application also provides an embodiment of a training device for an object posture recognition model. The device embodiment corresponds to the training method embodiment of the posture recognition model described above, and the device can be specifically applied to a controller.
[0135] The training device of the object posture recognition model of this embodiment includes:
[0136] The sample acquisition module is used to obtain the current training sample.
[0137] The iterative training module is used to iteratively train the object pose recognition model based on the current training samples.
[0138] In order to solve the above technical problems, the present application embodiment also provides a controller, such as: Figure 8 Computer equipment shown.
[0139] The computer device can be a terminal or a server.
[0140] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in this application.
[0141] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0142] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 6. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the object posture recognition method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.
[0143] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the object posture recognition method.
[0144] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.
[0145] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores a posture recognition program, and the posture recognition program can be executed by at least one processor so that the at least one processor performs the steps of the above-mentioned object posture recognition method.
[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. An object pose recognition model, It is characterized in that The model includes: an image segmentation module, a mask mapping module, a feature extraction module, a correlation calculation module, a coding matching module and a posture obtaining module; The image segmentation module is used to segment the object image to obtain the mask image of the object and the region of interest of the object; The mask mapping module is used to map the mask image of the object to a continuous space to generate a query vector; The feature extraction module is used to extract a feature map of an object in an object image based on a region of interest of the object; The correlation calculation module is used to perform correlation calculation on the query vector and the feature map of the object to obtain the encoding of the pixel coordinates of the object; The coding matching module is used to match the coding of the pixel coordinates of the object with the 3D coding of the vertices of the object model to obtain the corresponding relationship between the 2D pixel points and the 3D points of the object; The posture obtaining module is used to obtain the object posture in combination with the corresponding relationship; wherein, The posture obtaining module comprises: A feature extraction submodule, for extracting 2D features of an object based on a feature map of the object; An information obtaining submodule, used to obtain feature information based on the 2D feature: the feature information includes: weight distribution of 2D feature points, 2D point coordinates and 3D vertex coordinates; The PnP submodule is used to obtain the object posture by combining the corresponding relationship and the feature information.
2. A method for object posture recognition, It is characterized in that The method comprises the following steps: Acquire an object image; The object image is used as input of the object posture recognition model described in claim 1, and the object posture is obtained as output.
3. A posture recognition device, It is characterized in that The device comprises: An image acquisition module, used for acquiring an image of an object; The posture obtaining module is used to use the object image as the input of the object posture recognition model according to claim 1, and output the object posture.
4. A posture recognition system, It is characterized in that The system comprises: an image sensor and a controller; The image sensor is communicatively connected with the controller; The controller is used to implement the object posture recognition method described in claim 2.
5. A controller comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the object posture recognition method described in claim 2 are implemented.
6. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the object posture recognition method according to claim 2 are implemented.
