Image recognition method

Abstract

The present invention relates to an image recognition method, which comprises the following steps: inputting an image into a detection model to obtain a heat map tensor, a reference depth tensor, a weight tensor, and a sub-target tensor; obtaining K position index values from the heat map tensor; obtaining a fusion tensor based on the weight tensor and the sub-target tensor; obtaining a predicted depth tensor based on the fusion tensor and the reference depth tensor; referring to the K position index values, extracting K vectors from the predicted depth tensor; performing a transformation of a projection matrix on the K vectors to obtain K coordinate vectors in the real space. The method of the present invention can complete two tasks simultaneously through a one-time inference, namely detecting an object and detecting sub-targets included in the object, without building a model based on individual tasks, thereby reducing the latency on the consumer terminal.

Description

Technical Field

[0001] The present invention relates to an object tracking algorithm, and more particularly to an image recognition method. Background Art

[0002] Research and applications related to gestures and hand postures are a way of communicating with a computer system. With the development of computer vision technologies such as augmented reality (AR), virtual reality (VR), and large-screen display systems, applications related to hands on the market have gradually evolved from the previous hand gesture recognition towards hand pose estimation and tracking. Compared with simply recognizing gestures, if the state of the entire hand can be known, such as the position of each finger joint, more natural and fluent operations can be performed using both hands, and the application scope can be further expanded.

[0003] Generally speaking, a traditional hand pose tracking system needs to go through at least two stages of model processing, namely, a hand detection model and a finger joint detection model. First, use the hand detection model to detect the hand positions in each image. Then, use the finger joint detection model to calculate the actual positions of the finger joints of each hand in two-dimensional or three-dimensional space. After that, send the results to the system for subsequent identification or operation actions.

[0004] However, due to the increasing requirements of computer vision technologies, which require both real-time performance and high frame rate (Frames per second, FPS) analysis and identification. Therefore, the existing two-stage hand pose tracking system may cause high latency and reduce the user experience (Quality of Experience, QoE) of users, and its process also involves some complex pre-processing or post-processing, making it difficult to be applied to consumer terminals such as mobile phones or VR / AR glasses.

[0005] The "Background Art" section is only used to help understand the content of the present invention. Therefore, the content disclosed in the "Background Art" section may include some prior arts that are not known to those of ordinary skill in the art. The content disclosed in the "Background Art" section does not represent the problems that have been known or recognized by those of ordinary skill in the art before the filing of the present invention application, whether in this content or in one or more embodiments of the present invention. Summary of the Invention

[0006] The present invention provides an image recognition method that can find the positions of sub-objects included in a target object in an image through one stage.

[0007] The image recognition method according to an embodiment of the present invention includes: inputting an image into a detection model to obtain a heatmap tensor, a reference depth tensor, a weight tensor, and a sub-target tensor; obtaining K position index values from the heatmap tensor; obtaining a fusion tensor based on the weight tensor and the sub-target tensor; obtaining a predicted depth tensor based on the fusion tensor and the reference depth tensor; referring to the K position index values, extracting K vectors from the predicted depth tensor; and performing a transformation of a projection matrix on the K vectors to obtain K coordinate vectors in real space. Here, the heatmap tensor includes a plurality of probability values for the occurrence of a target object in a plurality of blocks corresponding to a plurality of position index values for predicting the image, and the target object includes a plurality of sub-targets. The reference depth tensor includes a first depth value corresponding to each block, which is the distance predicted between the imaging device capturing the image and each block. The weight tensor includes a plurality of weights for optimizing the sub-targets. The sub-target tensor includes a plurality of coordinate positions for predicting the sub-targets in the image and a second depth value of the sub-targets. The fusion tensor includes a plurality of fusion depth values obtained based on the weights and the second depth values. The predicted depth tensor includes a plurality of predicted depth values obtained based on the fusion depth values and the first depth values.

[0008] Using the method of the above embodiment, two tasks can be completed simultaneously through a single inference, namely, detecting the target object and detecting the sub-targets included in the target object, without building a model based on individual tasks. Description of the Drawings

[0009] Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention.

[0010] Figure 2 is a flowchart of an image recognition method according to an embodiment of the present invention.

[0011] Figure 3 is an architecture diagram of an image recognition model according to an embodiment of the present invention.

[0012] Figure 4 is a schematic diagram of finger joints of a hand according to an embodiment of the present invention.

[0013] Figure 5A and Figure 5B is a schematic diagram of a detection result according to an embodiment of the present invention.

[0014] Description of the Reference Numerals

[0015] 100: Electronic device

[0016] 110: Processor

[0017] 120: Memory

[0018] 300: Image

[0019] 310: Detection model

[0020] 320: Heat map tensor

[0021] 330: Reference depth tensor

[0022] 340: Weight tensor

[0023] 350: Sub - target tensor

[0024] 360: Position index list

[0025] 370: Fusion tensor

[0026] 380: Predicted depth tensor

[0027] 390: Target list

[0028] J01 - J21: Finger nodes

[0029] S205 - S230: Steps of the image recognition method Detailed implementation manners

[0030] Regarding the foregoing and other technical contents, features and effects of the present invention, they will be clearly presented in the following detailed description of the preferred embodiments in conjunction with the accompanying drawings. The directional terms mentioned in the following embodiments, such as: up, down, left, right, front or back, etc., are only with reference to the directions of the accompanying drawings. Therefore, the directional terms used are for illustration and not for limiting the present invention.

[0031] The present invention proposes an image recognition method, which can be implemented by an electronic device. To make the content of the present invention clearer, the following specific examples are given as examples that the present invention can actually be implemented based on.

[0032] Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention. Please refer to Figure 1 , the electronic device 100 includes a processor 110 and a memory 120. The processor 110 is coupled to the memory 120.

[0033] The processor 110 can be hardware with computing and processing capabilities (such as a chipset, a processor, etc.), software components (such as an operating system, an application program, etc.), or a combination of hardware and software components. For example, the processor 110 is a central processing unit (CPU), a graphics processing unit (GPU), or other programmable microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), or other similar devices.

[0034] The memory 120 is, for example, any type of fixed or removable random access memory, read-only memory, flash memory, secure digital card, hard disk, or other similar devices or a combination of these devices. A plurality of program code segments are stored in the memory 120, and after being installed, the above program code segments are executed by the processor 110 to thereby execute the display image recognition method.

[0035] Figure 2 It is a flowchart of an image recognition method according to an embodiment of the present invention. Figure 3 It is an architecture diagram of an image recognition model according to an embodiment of the present invention. The image recognition model of this embodiment is a one-stage neural network (NN) model. The input of the image recognition model is a two-dimensional image 300 of any type, and the output target list 390 includes a plurality of sub-target combinations ranked according to probability values.

[0036] Please refer to Figure 2 and Figure 3 , in step S205, the input image 300 is input to the detection model 310 to obtain a heat map tensor 320, a reference depth tensor 330, a weight tensor 340, and a sub-target tensor 350. Here, the tensor dimension of the image 300 is, for example, [H, L, C]. Among them, H is the height of the image, L is the length of the image, and C is the number of channels of the image. For example, if the input source is a color image (RGB-based Image), then C = 3. If the input source is a depth image (depth-based Image), then C = 1.

[0037] The heatmap tensor 320 includes a plurality of probability values for predicting the occurrence of a target object in a plurality of blocks corresponding to a plurality of position index values of the image 300. The target object further includes a plurality of sub-targets. The reference depth tensor 330 includes a first depth value (as a reference depth) corresponding to each block of the image 300. The first depth value is the distance between the imaging device that captures the image 300 and each block. The weight tensor 340 includes a plurality of weights for optimizing the plurality of sub-targets. The sub-target tensor 350 includes coordinates for predicting the position of each sub-target in the image 300 and a second depth value corresponding to each sub-target.

[0038] The detection model 310 is a feature extractor based on a Convolutional Neural Network (CNN). The architecture of the detection model 310 is partially similar to the YOLO Fourth Edition (YOLOv4) algorithm. The detection model 310 is a model architecture with a single input and multiple outputs, and the tensors of the multiple outputs are all reduced by an integer factor S. For example, for the resolution of the image 300 being H×L, the resolutions of the obtained heatmap tensor 320, reference depth tensor 330, weight tensor 340, and sub-target tensor 350 are all H / S×L / S.

[0039] If the device source of the input (image 300) is a color imaging device (color camera), a dataset of color images is used to train the detection model 310. If the device source of the input (image 300) is a depth imaging device, a dataset of depth images is used to train the detection model 310. Each dataset contains the three-dimensional positions of a plurality of target objects and the projection matrix of the imaging device.

[0040] Here, the target object to be detected is a hand, and the sub-targets are the finger joints of the hand. Figure 4 It is a schematic diagram defining the finger joints of a hand according to an embodiment of the present invention. The definition of the finger joints of a hand can be, for example, Figure 4 the 21 finger joints J01 to J21 shown. Using the image recognition model of this embodiment, K hands and their respective 21 finger joints can be detected in the image 300.

[0041] The heatmap tensor 320 includes probability values for predicting the occurrence of a hand, the reference depth tensor 330 includes the distance (first depth value) between the imaging device that captures the image 300 and the hand, the weight tensor 340 includes weights for optimizing the finger joints, and the sub-target tensor 350 includes the coordinate positions of each finger joint in the image 300 and a second depth value corresponding to each finger joint. The second depth value corresponding to each finger joint refers to the distance from each finger joint to the wrist.

[0042] The tensor dimension of the heat map tensor 320 is [H / S, L / S, 2]. Among them, the first and second dimensions represent the position index values (i, j) of the blocks, where i = {1, 2,..., H / S} and j = {1, 2,..., L / S}, and the third dimension "2" represents the probability values corresponding to the occurrence of two types of target objects (i.e., "left hand" and "right hand") for each position index value (i, j). That is, the image 300 is input into the detection model 310 and sliced into blocks of size H / S × L / S, and two probability values are estimated for each block, namely, the probability value of the occurrence of the left hand and the probability value of the occurrence of the right hand. Therefore, the heat map tensor 320 includes H / S × L / S × 2 block data. The probability value is between 0 and 1.

[0043] The tensor dimension of the reference depth tensor 330 is [H / S, L / S, 1]. Among them, the first and second dimensions represent the position index values (i, j) of the blocks, and the third dimension "1" represents that each block corresponding to the position index value (i, j) corresponds to 1 first depth value. The reference depth tensor 330 includes H / S × L / S × 1 first depth values.

[0044] The tensor dimension of the weight tensor 340 is [H / S, L / S, N]. Among them, the first and second dimensions represent the position index values (i, j) of the blocks, and the third dimension "N" represents the weight values for optimization corresponding to the N finger joints included in each block corresponding to the position index value (i, j). The weight tensor 340 includes H / S × L / S × N weight values.

[0045] The tensor dimension of the sub-goal tensor 350 is [H / S, L / S, N, 3]. Among them, the first and second dimensions represent the position index values (i, j) of the blocks, the third dimension "N" represents that each block corresponding to the position index value (i, j) corresponds to N finger joints, and the fourth dimension "3" represents the coordinate positions of each finger joint in x, y, and z. The sub-goal tensor 350 includes H / S × L / S × N groups of coordinate positions (x, y, z), where x and y represent the positions of the finger joints in the image, and z represents the depth value of the finger joint (i.e., the second depth value).

[0046] Next, in step S210, K position index values are obtained from the heat map tensor 320. For example, among the H / S × L / S × 2 block data included in the heat map tensor 320, starting from the block data with the highest probability value, the K position index values corresponding to the K block data are taken and recorded in the position index list 360. Among them, K is the number of target objects (e.g., hands). For example, the position index list 360 records: position index values (gx_1, gy_1), (gx_2, gy_2),..., (gx_K, gy_K).

[0047] In step S215, based on the weight tensor 340 and the sub-goal tensor 350, the fusion tensor 370 is obtained. Here, the weight tensor 340 and the sub-goal tensor 350 are convolved using the following formula to obtain the fusion tensor 370. The fusion tensor 370 includes a plurality of fusion depth values obtained based on the weights and the second depth values.

[0048]

[0049] Among them, ks is the kernel size, W is the weight tensor 340, V is the sub-goal tensor 350, a = {1, 2,..., H / S}, b = {1, 2,..., L / S}, c = {1, 2,..., N}, N is the number of sub-goals (i.e., the number of nodes), d = {1, 2, 3} (representing the x, y, and z axes). O(a, b, c, d) is the fusion tensor 370. The tensor dimension of the fusion tensor 370 is [H / S, L / S, N, 3]. The fourth dimension "3" represents the coordinate positions of each finger joint predicted on the x, y, and z axes, and the depth value corresponding to z is the fused depth value after convolution.

[0050] After that, in step S220, based on the fusion tensor 370 and the reference depth tensor 330, the predicted depth tensor 380 is obtained. The predicted depth tensor 380 includes a plurality of predicted depth values obtained based on the fused depth values and the first depth values. Specifically, the fused depth value corresponding to each position index value in the fusion tensor 370 (i.e., the z value in the fourth dimension of the fusion tensor 370) is added to the first depth value corresponding to each position index value in the reference depth tensor 330 (i.e., the value in the third dimension of the reference depth tensor 330) to obtain the predicted depth tensor 380. This is because the predicted depth value from the imaging device to the finger joint will be the sum of the distance between the imaging device and the hand (the first depth value) and the distance between each finger joint and the wrist (the fused depth value).

[0051] Finally, in step S225, referring to the position index values, K vectors are taken out from the predicted depth tensor 380. According to the position index values recorded in the position index list 360 obtained from the heatmap tensor 320, the corresponding K vectors are taken out from the predicted depth tensor 380, and then the target list 390 is obtained. Each vector records the positions of N finger joints. For example, the target list 390 includes vectors (J_1_1, J_1_2,... J_1_N), vectors (J_2_1, J_2_2,... J_2_N), …, vectors (J_K_1, J_K_2,... J_K_N).

[0052] For the first position index value (gx_1, gy_1) of the position index list 360, its corresponding vector is (J_1_1, J_1_2,...J_1_N), and "J_1_1", "J_1_2", …, "J_1_N" respectively represent the positions of the N finger joints of the position index value (gx_1, gy_1). For the second position index value (gx_2, gy_2) of the position index list 360, its corresponding vector is (J_2_1, J_2_2,...J_2_N), and "J_2_1", "J_2_2", …, "J_2_N" respectively represent the positions of the N finger joints of the position index value (gx_2, gy_2). For the K-th position index value (gx_K, gy_K) of the position index list 360, its corresponding vector is (J_K_1, J_K_2,...J_K_N), and "J_K_1", "J_K_2", …, "J_K_N" respectively represent the positions of the N finger joints of the position index value (gx_K, gy_K).

[0053] Figure 5A and Figure 5B is a schematic diagram of the detection result according to an embodiment of the present invention. Figure 5A The detection result of one hand is shown. Figure 5B The detection results of two hands are shown. By the above method, one or more finger joints of the hand can be accurately detected in the image.

[0054] After that, in step S230, the K vectors are subjected to the conversion of the Projection Matrix to obtain K coordinate vectors in the real space. Through the above steps, the hand posture appearing on the input image 300 can be tracked.

[0055] In summary, the embodiments of the present invention can simultaneously complete two tasks through a single inference, namely detecting the target object and detecting the sub-targets included in the target object, without building models based on individual tasks. Accordingly, when the embodiments of the present invention are applied to multi-hand posture tracking, any type of image input can output multiple finger joint combinations on the image ranked according to probability values.

[0056] In addition, as long as the embodiments of the present invention know that the input source is one of the types of color images and depth images, they can select the same type of data set to retrain the model according to the type of the input source. Without changing the CNN model architecture, the architecture used in the embodiments of the present invention can still complete hand detection and finger joint regression at one time.

[0057] Since the intermediate process of the embodiments of the present invention does not require sub-images extracted by the bounding boxes of object detection, there will be no problem of capturing poor sub-images and thus reducing the finger joint estimation accuracy. In the case where there are K hands in an image, the traditional multi-hand pose tracking system needs to perform K + 1 times of model operations. On the contrary, the embodiments of the present invention can obtain the positions of K hands and their finger joints simultaneously after 1 operation. Therefore, the embodiments of the present invention can reduce the latency on the consumer terminal and improve the quality of the user experience.

[0058] The above are only the preferred embodiments of the present invention, and the scope of implementation of the present invention cannot be limited thereby. All simple equivalent changes and modifications made according to the claims and the content of the specification of the present invention still fall within the scope covered by the present invention. In addition, the technical solutions of any embodiment or claim of the present invention do not have to achieve all the purposes, advantages or features of the present invention. In addition, the abstract part and the title are only used to assist in searching patent documents and are not used to limit the scope of rights of the present invention. In addition, the terms "first", "second", etc. mentioned in this specification or claims are only used to name elements or distinguish different embodiments or scopes, and are not used to limit the upper or lower limits of the number of elements.

Claims

1. An image recognition method, characterized in that Including: Inputting an image into a detection model to obtain a heatmap tensor, a reference depth tensor, a weight tensor, and a sub-target tensor, where the heatmap tensor includes multiple probability values of the occurrence of a target object in multiple blocks corresponding to multiple position index values for predicting the image, the target object includes multiple sub-targets, the reference depth tensor includes a first depth value corresponding to each block among the multiple blocks, the first depth value is the distance between the imaging device that captures the image and each block, the weight tensor includes multiple weights for optimizing the multiple sub-targets, and the sub-target tensor includes multiple coordinate positions of the multiple sub-targets in the image and multiple second depth values of the multiple sub-targets; Obtaining K position index values from the heatmap tensor; Based on the weight tensor and the sub-target tensor, obtaining a fusion tensor, where the fusion tensor includes multiple fusion depth values obtained based on the multiple weights and the multiple second depth values; Based on the fusion tensor and the reference depth tensor, obtaining a predicted depth tensor, where the predicted depth tensor includes multiple predicted depth values obtained based on the multiple fusion depth values and the multiple first depth values; Referring to the K position index values, extracting K vectors from the predicted depth tensor; and Performing a transformation of a projection matrix on the K vectors to obtain K coordinate vectors in real space.

2. The image recognition method according to claim 1, wherein The heatmap tensor includes multiple block data corresponding to the multiple blocks, and each block data among the multiple block data includes a corresponding position index value and two probability values, and the two probability values represent the probability value that the corresponding block includes a left hand and the probability value that it includes a right hand, where the step of obtaining K position index values from the heatmap tensor includes: According to the two probability values, starting from the block data with the highest probability value among the multiple block data, extracting the K position index values corresponding to the K block data.

3. The image recognition method according to claim 1, characterized in that, The resolution of the image is H×L. After inputting the image into the detection model, the heatmap tensor, the reference depth tensor, the weight tensor, and the sub-target tensor with a resolution reduced by S times are obtained, The step of obtaining the fusion tensor based on the weight tensor and the sub-target tensor includes: Convolving the weight tensor and the sub-target tensor using the following formula: Among them, O(a, b, c, d) is the fusion tensor, and e and f are the variables of the summation symbol ∑ in the above formula. ks is the kernel size, W is the weight tensor, V is the sub-goal tensor, a = {1, 2,..., H / S}, b = {1, 2,..., L / S}, c = {1, 2,..., N}, N is the number of sub-goals, and d = {1, 2, 3}.

4. The image recognition method according to claim 1, wherein The step of obtaining the predicted depth tensor based on the fusion tensor and the reference depth tensor includes: Adding the multiple fusion depth values corresponding to each position index value in the fusion tensor to the first depth value corresponding to each position index value in the reference depth tensor respectively to obtain the multiple predicted depth values corresponding to each position index value.

5. The image recognition method according to claim 1, wherein The detection model is a feature extractor based on a convolutional neural network.

6. The image recognition method according to claim 1, wherein The target object is a hand, and the multiple sub-targets are finger joints.

Images