Object and key point detection method and host and driver monitoring system thereof

Abstract

The invention relates to a method for detecting objects and key points. The method comprises the following steps of: receiving an image; executing a deep neural network architecture based on the image to obtain one or more object bounding boxes; executing a deep neural network architecture based on the one or more object bounding boxes to obtain relative positions of one or more key points corresponding to the one or more object bounding boxes; and outputting the positions of the one or more object bounding boxes and the one or more key points.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection and recognition, and in particular to an object and key point detection method and a host and driver monitoring system thereof. Background Art

[0002] A driver image monitoring system is a system used to monitor and evaluate driver behavior. Its input is usually an image captured by a color photography wide-angle lens. The pixels of the above image are usually encoded in three colors: red, green, and blue. The image is detected using a neural network image to determine whether the driver has violated the law or is inattentive.

[0003] In order to detect whether the driver is holding a distracting object, such as a mobile phone or a cigarette, the above-mentioned driver image monitoring system will perform more than one type of object detection. In addition, in order to detect whether the driver is concentrating, the above-mentioned driver image monitoring system will perform key point detection. The above-mentioned key points may include key points of facial and hand features. In short, the above-mentioned driver image monitoring system requires object detection function and key point detection function. Therefore, the above-mentioned driver image monitoring system requires two or more deep learning-like neural network models with different architectures to respectively implement object detection and key point detection functions, which requires more than twice the judgment time.

[0004] In addition, since ordinary visible light band lenses cannot capture clear images in low-light environments. In other words, the images lack details for identification. The neural learning network model in deep learning cannot detect object features and key point features. In this case, in order to detect key points, the first neural model must be used to detect the position and size of the object to which the key points belong, and the detected object image is cropped, and then the second neural model is used to detect the position of each key point. When the first model detects N objects (where N is a positive integer), the second model must also perform N detections. These calculation procedures will cause huge time costs on the system, and will also incur a lot of time costs in post-processing. Due to the high speed of the vehicle, if it takes a long time to detect that the driver is not paying attention, there may not be enough time to warn the driver.

[0005] Based on this, a method that can reduce the detection time and speed up the detection is urgently needed. Through a deep learning-like neural network model, the key point detection task can be achieved without the above two-step judgment. In other words, one of the above two-step judgment can be omitted, and the implementation of one-step judgment can achieve the dual functions of object detection and key point detection. Summary of the invention

[0006] According to an embodiment of the present invention, a method for detecting objects and key points is provided, comprising: receiving an image; executing a deep neural network architecture based on the image to obtain one or more object bounding boxes; executing a deep neural network architecture based on the one or more object bounding boxes to obtain relative positions of one or more key points corresponding to the one or more object bounding boxes; and outputting the positions of the one or more object bounding boxes and the one or more key points.

[0007] Preferably, in order to provide an image with more details under low light conditions, the image contains information in the infrared band.

[0008] Preferably, in order to implement a deep neural network architecture using a neural network, wherein:

[0009] The deep neural network architecture includes: a backbone architecture network; a neck network, which includes a feature pyramid network and a pyramid attention network to extract features from the backbone architecture network; and a detection head, which obtains one or more object bounding boxes and the positions of one or more key points from the neck network.

[0010] Preferably, in order to identify objects and key points in shallow and deep information, the detection head includes: a large object detection head, a medium object detection head and a small object detection head, which are used to obtain the positions of one or more object bounding boxes and one or more key points from multiple blocks of different sizes in the neck network respectively.

[0011] Preferably, in order to determine whether the object bounding box is credible when executing the deep neural network architecture, the deep neural network architecture is trained based on multiple candidate samples and multiple reference real samples. The threshold used to determine whether the predicted box is an object bounding box corresponds to the average and squared difference of the multiple candidate samples and their corresponding reference real samples.

[0012] Preferably, in order to train the object detection performance of the deep neural network architecture, the object box regression loss function of the deep neural network structure is related to the following two: the above-mentioned threshold; and the ratio of multiple intersections to unions between multiple candidate samples and their corresponding multiple benchmark real samples.

[0013] Preferably, in order to train the object key point detection performance of the deep neural network architecture, the key point loss function of the deep neural network architecture is a wing loss function.

[0014] Preferably, in order to make the deep neural network architecture have better key point detection performance, during the training process of the deep neural network architecture, one or any combination of the following algorithms is used to evaluate the detection performance of object key points during training: object key point similarity (OKS, Object Key-point Similarity) algorithm; and correct key point percentage (PCK, Percentage of Correct Key-points) algorithm.

[0015] Preferably, in order to enable the deep neural network architecture to provide the driver status within a vehicle (such as a car), one or more object bounding boxes and their corresponding one or more key point systems correspond to one or any combination of the following categories of objects: human face, human hand, mobile phone, cigarette, glasses and seat belt.

[0016] An embodiment of the present invention provides a host for object and key point detection, including: one or more processors, configured to execute a plurality of computer instructions stored in a non-volatile memory to implement the above-mentioned object and key point detection method.

[0017] One embodiment of the present invention provides a driver monitoring system for object and key point detection, including a host disposed in a vehicle (such as a car); and a photographic device for providing images.

[0018] In summary, the object and key point detection method provided by the present invention, as well as the host and driver monitoring system thereof, can realize the dual functions of object detection and key point detection through one-step judgment, reduce post-processing time, solve the problems caused by environmental and light changes, realize less processing and calculation requirements, reduce calculation time, and improve the real-time performance of the driver monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 and Figure 2 is a schematic diagram of a deep neural network architecture 100 according to an embodiment of the present invention.

[0020] Figure 3 FIG. 4 is a schematic diagram of an adaptive training sample selection procedure according to an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of the distance between the predicted box and the real box of the present invention.

[0022] Figure 5 is a diagram of the intersection and union of two boxes.

[0023] Figure 6FIG. 4 is a schematic diagram of a look-up table (LUT) of a loss function according to an embodiment of the present invention.

[0024] Fig. 7A and 7B They are schematic diagrams of the results of object detection and key point detection based on the deep neural network architecture 100 provided by the present invention.

[0025] Figure 8 FIG. 8 is a block diagram of a driver monitoring system 800 according to an embodiment of the present invention.

[0026] Fig. 9 is a flowchart of an object and key point detection method 900 according to an embodiment of the present invention.

[0027] Description of reference numerals:

[0028] 100: deep neural network architecture; 110: input image; 120: backbone architecture network; 130: neck network; 140: detection head; 141: large object detection head; 142: medium object detection head; 143: small object detection head; 150: output image.

[0029] 410: predicted box; 420: true box; 430: minimum closure area or minimum union rectangular area.

[0030] 800: driver monitoring system; 810: host; 811: processor; 812: sub-processor; 813: peripheral device connection device; 814: storage device; 820: photography device.

[0031] 900: Object and key point detection method; 910-940: Steps. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only a part of the present invention, rather than a comprehensive collection of embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0033] The terms "first", "second", "third", etc. (if any) in the description, claims and drawings of the patent application are used to distinguish similar objects and do not need to be used to describe a specific order or sequence. It should be understood that the objects described can be interchanged where appropriate. In the description of the present invention, "plurality" means two or more, unless otherwise specifically limited. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusiveness. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware circuits or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] In the description of the present invention, it should be noted that, unless otherwise specified and limited, the terms "installed", "connected" and "connected" should be understood broadly, for example, they can be fixedly connected, detachably connected or integrated; they can be mechanically connected, electrically connected or mutually communicated; they can be directly connected or indirectly connected through an intermediate medium, and two elements can be internally connected or have an interactive relationship between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to schematic diagrams and specific implementation methods.

[0036] See also Figure 1 and Figure 2 , which are schematic diagrams of a deep neural network architecture 100 of an embodiment of the present invention. This deep neural network architecture 100 is a single architecture for integrating object detection and key point detection, and can simultaneously implement two different types of detection and recognition: object detection and key point detection. The deep neural network architecture 100 is based on the foundation of the object detection architecture, but is extended to the function of key point detection. The difference from the previous technology is that the previous key point detection uses the absolute position of the entire image for prediction, while the deep neural network architecture 100 uses the object box center point regression to calculate the relative position of each key point.

[0037] In one application of the present invention, it is applicable to a driver monitoring system. The above-mentioned deep neural network architecture 100 is used to detect objects including faces, hands, mobile phones, cigarettes, glasses and / or seat belts, and to identify the key points of the above-mentioned types of objects. It can be understood by those of ordinary skill in the art that although the present invention can be applied to driver monitoring systems, it can also be applied to embodiments of object recognition and key point recognition of other objects. For example, in certain surgeries, the deep neural network architecture 100 provided in the present invention can be used to detect objects such as certain organs, surgical instruments, cotton, and detect key points of certain organs and surgical instruments. In an embodiment of missile seeker image recognition, the deep neural network architecture 100 provided in the present invention can be used to detect objects such as tanks, armored vehicles or other tactical vehicles, and to detect key points of weaknesses such as the top covers of certain tanks.

[0038] like Figure 1 and Figure 2 As shown, the deep neural network architecture 100 includes a three-layer network structure, where the image 110 is input into the backbone network 120, and then after multiple down-sampling, it is input into the neck network 130 for feature extraction, and finally output to the detection head 140. The detection head 140 outputs detection results such as the bounding box and key points of the object. These detection results can be superimposed on the original input image 110 to form an output image 150.

[0039] In one embodiment, the backbone architecture network 120 may include the following or one of their derivatives: Darknet53, DenseNet, CSPDenseNet (Cross Stage Partial Dense Network), etc., for acceleration and parallel computing.

[0040] In one embodiment, the neck network 130 may include a network of a feature pyramid network (FPN) and a pyramid attention network (PAN), so that deep and shallow information can be exchanged in the architecture. In another embodiment, the neck network 130 may combine two models: a spatial pyramid pooling layer (SPP) and a path aggregation network (PANet).

[0041] The detection head 140 can detect objects. In the present invention, the detection head 140 adds a key point detection branch to output a detection result with an object bounding box and key points. It can be understood by those skilled in the art that the backbone architecture network 120 and the neck network 130 can be implemented using other types of deep neural networks as long as they can enable the subsequent detection head 140 to obtain the bounding box and key point detection results of large, medium and small objects.

[0042] like Figure 1 and Figure 2 As shown, the detection head 140 may include three different branches, each of which detects large objects, medium objects, and small objects based on different orders of the pyramids in the neck network 130. In each detection head 141, 142, and 143, the image is divided into a plurality of blocks, each of which has a plurality of reference frames (anchors). The detection heads 141, 142, and 143 can predict the offset of the target object relative to the reference frame, and each reference frame can predict one or more bounding boxes. Finally, the bounding boxes with low probability are eliminated according to the trust index.

[0043] When training the detection head 140, it is necessary to first perform a pre-processing procedure on the training data. In one embodiment, an Adaptive Training Sample Selection (ATSS) procedure can be used to calculate the location of the positive samples of the objects to be detected. Figure 3 , Figure 3 A schematic diagram of an adaptive training sample selection procedure provided by an embodiment of the present invention. Figure 3 The image shown is divided into a plurality of rectangular blocks (units). The size of the block is determined by the input image 110 of the deep neural network architecture 100 and the number of downsampling times. When the input image 110 of the deep neural network architecture 100 is 320×320 pixels and undergoes five downsampling times, the size of each rectangular block is 10×10 pixels. Figure 3 The output size of the detection head shown is [n, h*w, c], where h is the height of the matrix in the block matrix, w is the width of the matrix in the block matrix, h*w is the number of blocks, c represents the number of key points, and n represents the number of channels in the deep neural network architecture 100.

[0044] Please refer to Table 1, which is a calculation formula for determining the thresholds of positive samples and negative samples in the above-mentioned adaptive training sample selection procedure. Table 1 contains three equations. Equation (1) is used to calculate the average value or average value μ of the candidate sample and the reference truth (Ground True) sample. Equation (2) is used to calculate the variance or square difference σ between the candidate sample and the reference truth (Ground True) sample. Equation (3) is used to calculate the threshold IoU Threshold of the positive sample and the negative sample, that is, the sum of the mean or average value μ obtained by equation (1) and the variance or square difference σ obtained by equation (2). In one embodiment, the threshold IoU Threshold used to determine the positive sample and the negative sample can be related to the mean or average value μ and the variance or square difference σ, for example, the threshold IoU Threshold can be proportional to the mean or average value μ. The threshold IoU Threshold can also be proportional to the variance or square difference σ.

[0045]

[0046] Table 1. Threshold calculation of target sampling for adaptive training sample selection procedure

[0047] like Figure 3 As shown, the yellow block represents the positive sample block. Next, based on the reference coordinates of the rectangular block, the relative position or relative coordinates of each key point and the reference coordinates are calculated. In one embodiment, the reference coordinates of the rectangular block generally refer to the coordinates of one of the four corners, such as the upper left corner. Figure 3 As shown, the relative position or relative coordinates between the bottom red key point and the block can be calculated. As mentioned above, this is one of the differences between the present invention and the prior art.

[0048] The loss functions used in the deep neural network architecture 100 are shown in Table 2:

[0049]

[0050] Table 2 Loss functions for target and key point detection

[0051] Table 2 contains three equations. Equation (4) is used to calculate the loss function (GFL, Generalized Focal Loss) of object detection, which is the sum of the loss function value of the category confidence (QFL, Quality Focal Loss) and the object bounding box regression loss function value (DFL, Distribution Focal Loss). Where p yl Represents the test result, p yris the real result. In one embodiment, the loss function GFL used to calculate object detection can be related to QFL and DFL, but not necessarily in a relationship of and. For example, the loss function GFL can be proportional to QFL. The loss function GFL can also be proportional to DFL. Explained in a physical way, assuming that the deep neural network architecture 100 is a shooter, and the predicted target position is the position of the bull's eye in the shooting, the farther the position of the arrow is from the bull's eye, the more value is lost, and vice versa.

[0052] Please refer to Figure 4 , which is a schematic diagram of the distance between the predicted box and the real box of the present invention. Figure 4 The predicted box 410 is shown in the upper left corner, the true box 420 is shown in the lower right corner, and the minimum closure area or minimum union rectangular area 430 of the two. The predicted box 410 has a center point b. The true box 420 has a center point b gt P is the Euclidean distance between the two center points. C represents the diagonal distance of the minimum closure area 430 that can contain both the predicted box 410 and the true box 420.

[0053] Based on the above parameters, equation (5) can be used to calculate the object box regression loss function (DIUO, Distance IOUloss). The object box regression loss function is related to the ratio of intersection over union (IoU, Intersection Over Union). Please refer to Figure 5 , which is a diagram of the intersection and union of two boxes. Figure 4 The predicted box 410 and the true box 420 in can be considered as Figure 5 When the intersection above the graph and the union below the graph are closer to each other, it means that the predicted frame 410 is closer to the true frame 420. When the intersection above the graph is equal to the union below the graph, that is, when the ratio of intersection to union IoU is 1, it means that the predicted frame 410 is equivalent to the true frame 420. Please refer to equation (7) in Table 3, which is the calculation method of the above intersection to union ratio.

[0054]

[0055] Table 3 Calculation method of the ratio of intersection to union

[0056] The subtrahend shown in equation (5) is the square of the Euclidean distance b between the center of the predicted box 410 and the center of the true box 420 2 Distance from diagonal line c 2Similarly, the closer the predicted box 410 is to the true box 420, the closer this ratio will be to 0. In other words, the regression loss function of the object box will be closer to 1. Explained in a physical way, the farther the target box predicted by the deep neural network architecture 100 is from the true box, the more values ​​are lost; otherwise, the fewer values ​​are lost. In one embodiment, the object box regression loss function DIOU may be proportional to the ratio of the intersection over the union IoU. In one embodiment, the object box regression loss function DIOU may be inversely proportional to the subtrahend of equation (5). In one embodiment, the object box regression loss function DIOU may be proportional to the square of the Euclidean distance b between the center of the predicted box 410 and the center of the true box 420 2 Inversely proportional.

[0057] Equation (6) shows the loss function of key point detection (Wing loss function). Please refer to Figure 6 , which is a schematic diagram of a lookup table (LUT, Look Up Table) of a loss function of an embodiment of the present invention. Because its shape is like a pair of spread wings, this loss function is called a wing loss function. The input value of the loss function is the difference or distance value between the detected key point pt and the true point gt. By substituting the difference into the lookup table, the error value can be obtained. Explained in a physical way, the greater the distance difference between the two, the greater the loss value. Conversely, the less the loss value. In one embodiment, the above-mentioned loss function can be a quadratic function.

[0058] When training the deep neural network architecture 100, the performance of object detection and key point detection can be evaluated separately. Figure 5 The ratio is calculated by using equation (7). The more the predicted box 410 overlaps with the true box 420, the closer the ratio is to 1.

[0059] Regarding the performance evaluation of key point detection, the object key point similarity (OKS) algorithm can be used. The accuracy of key point detection can be measured by comparing the distance between the predicted key point position and the actual key point position, similar to the distance function D in equation (6). It is a similarity measure based on Euclidean geometric distance. Considering the size of the actual key point and the importance of the corresponding part, each key point information includes the key point coordinates and a score value. This score value can be used to indicate the visibility or confidence of the key point. When evaluating, the prediction results are usually screened according to the weighted score of the key point, and the key points with higher scores are retained.

[0060] Please refer to equation (8) in Table 4, which is the above-mentioned object key point similarity (OKS, Object Key-pointSimilarity) algorithm. Among them, dp represents the Euclidean distance, Sp represents the scale factor of the current object, that is, the square root of the area occupied by the key point object in the real object, and σ represents the normalization factor of the key point, where the normalization factor is the standard deviation calculated based on the Ground True samples in all data sets, reflecting the degree of influence of the key point on the whole. The larger the value of the normalization factor, the worse the annotation effect of the key point in the entire data set. On the contrary, the smaller the value of the normalization factor, the better the annotation effect of the key point in the entire data set. v p i=1 indicates that the score or visibility of the key point is 1. Δ is a selection function used to select visible key points for calculation.

[0061]

[0062] Table 4 Object key point similarity evaluation formula

[0063] Match the real key points with the predicted key points and calculate the Euclidean distance between each matched key point. Then, based on the above Euclidean distance d p , the square root of the area occupied by the key point object in the real object Sp, the standard deviation normalization factor and the score value, calculate the similarity measure OKS p . Similarity measure OKS p The range is between 0 and 1. The closer it is to 1, the higher the match between the predicted result and the actual result.

[0064] Another algorithm for evaluating the performance of the key point detection model is the Percentage of Correct Key-points (PCK) algorithm. Similarly, the Euclidean distance between the key points predicted by the deep neural network architecture 100 and the real annotated key points is compared. If the distance between the two points is less than a certain threshold, it is considered to be a correct prediction. The threshold can be a certain ratio of the actual length or width. The above-mentioned ratio of correct key points is the percentage of correctly predicted key points. Please refer to equation (9) in Table 5, which is the evaluation formula for the correct key point ratio.

[0065]

[0066] Evaluation formula for the correct key point ratio in Table 5

[0067] As shown in equation (9), N is the number of key points, p i is the position of the i-th predicted key point, g iis the position of the key point of the i-th true annotation, ||||2 represents the Euclidean distance, and 1() is an indicator function or conditional function. When the condition in the brackets is true, the function value is 1, otherwise it is 0. As mentioned above, α can be a certain ratio of the true length or width, such as 0.1, 0.15 or 0.2, etc. S is a reference scale, such as the length of the object, such as the size of the head or the length of the palm.

[0068] Please refer to Fig. 7A and 7B , which are schematic diagrams of object detection and key point detection results based on the deep neural network architecture 100 provided in the present invention. In these two result diagrams, two categories of objects can be detected. The orange frame represents the hand, and the pink frame represents the face. In addition, multiple key points are detected in the hand frame, representing the joints of the hand. In other words, the deep neural network architecture 100 provided by the present invention can also provide information about the object category.

[0069] The advantage of the deep neural network architecture 100 provided by the present invention is that the functions of object detection and key point detection can be achieved by using a single model for one judgment. Please refer to Table 6 below. Compared with the previous technology, the present invention significantly reduces the judgment time, while the accuracy is only slightly reduced. After conversion, the overall cost performance is as high as 269 times, which does reduce the computational cost and time of judgment without affecting the detection accuracy.

[0070]

[0071] Table 6 Comparison between the present invention and the prior art

[0072] See also Figure 8 , Figure 8 800 is a block diagram of a driver monitoring system 800 according to an embodiment of the present invention. The driver monitoring system 800 can be implemented in a vehicle, which can be a mobile device that requires driver operation, such as a vehicle, a ship, a submarine, an aerospace equipment, etc. The vehicle can also include a driver's seat of any machine, such as a driver's seat of a crane. The above-mentioned deep neural network architecture 100 can be implemented in a host 810. The host 810 includes at least one processor 811 for executing an operating system to control the host 810 and / or the driver monitoring system 800. The processor 811 can be an x86, x64, ARM, RISC-V or other industrial standard instruction set processor. The operating system can be an operating system of the UNIX series, Windows series, Android series, iOS series or other series, or a real-time operating system.

[0073] The host 810 may include one or more co-processors 812 to accelerate the judgment of the deep neural network architecture 100. The one or more co-processors 812 may be a graphics processing unit (GPU), a neural network processor (NPU), an artificial intelligence processor (APU), and other processors with multiple vector logic and computing units to accelerate the judgment of the deep neural network architecture 100. The present invention does not limit the host 810 to have a co-processor 812 to implement the judgment of the deep neural network architecture 100.

[0074] The host 810 also includes a peripheral device connection device 813, which can be used to connect one or more camera devices 820. The host 810 can also include a storage device 814 for storing the above-mentioned operating system and a program for implementing the deep neural network architecture 100. Ordinary technicians in this field need to have knowledge of computer organization and architecture, operating system, system program, artificial intelligence and deep neural network. They can change or derive the embodiments of the above-mentioned host 810 and driver monitoring system 800, as long as they can implement the deep neural network architecture 100 provided in the present invention.

[0075] The camera device 820 can be connected to the peripheral device connection device of the host 810 through an industrial standard interface, such as common wired or wireless connection technology, such as UWB, WiFi, Bluetooth, USB, IEEE 1394, UART, iSCSI, PCI-E, SATA and other industrial standard technologies. The present invention is not limited to the above industrial standard interfaces, as long as the speed of providing images by the camera device 820 can meet the real-time requirements of the driver monitoring system 800.

[0076] The camera device 820 may include a sensor in a conventional visible light band or an infrared band. In low light conditions, the camera device 820 may also include an infrared light source for illuminating the driver's seat to be monitored, ensuring that the camera device 820 can output images with sufficient details for processing by the deep neural network architecture 100.

[0077] In one embodiment, the peripheral device connection device 813 can output the result of the judgment of the deep neural network architecture 100 to other devices or systems. The judgment of the above deep neural network architecture 100 includes more than one object bounding box and one or more key point positions associated with it. For example, the judgment result can be output to a storage device so as to find certain key objects when playing back surveillance images. For example, when playing back surveillance images, you can search for object bounding boxes with cigarettes and human faces, as well as images with cigarette filter key points close to the key points of the human mouth, in order to determine whether the driver or passenger is smoking. In another embodiment, search for object bounding boxes with mobile phones and human faces, as well as images with key points of mobile phones close to key points of the human mouth, in order to determine whether the driver or passenger is making a call.

[0078] In another embodiment, the host 810 may execute other programs to receive the judgment result of the deep neural network architecture 100. For example, when the judgment result meets certain conditions, a warning program may be executed so that other peripheral devices warn the driver.

[0079] In a further embodiment, the host 810 can send the judgment result to the driving control system of the vehicle through the peripheral device connection device 813, and the driving control system determines whether it is necessary to switch to the mode of the automatic driving control program. For example, when it is detected that the key points of the driver's hands leave the steering wheel, the driving control system can temporarily switch to the fourth or fifth level automatic driving mode. In other words, in the present invention, the host 810 can output the judgment result of the deep neural network architecture 100, or output information derived from the judgment conclusion of the deep neural network architecture 100.

[0080] See also Fig. 9 , Fig. 9 Flow chart of an object and key point detection method 900 provided in one embodiment of the present invention. The object and key point detection method 900 can be implemented based on the deep neural network architecture 100, in particular, by Figure 8 The object and key point detection method 900 may be implemented by computer instructions executed by the host computer 810. The object and key point detection method 900 may start at step 910.

[0081] Step 910: Receive an image. Figure 8 As shown, the host 810 may receive an image from the camera 820. Next, the process proceeds to step 920.

[0082] Step 920: Execute a deep neural network architecture (such as deep neural network architecture 100) based on the image to obtain one or more object bounding boxes. Next, the process proceeds to step 930.

[0083] Step 930: Execute a deep neural network architecture based on one or more object bounding boxes to obtain the relative position of one or more key points corresponding to the object bounding boxes. It is worth noting that the relative position of one or more key points corresponds to the center position or corner position of their object bounding boxes, rather than the position of the image. Based on this, it can be inferred that the deep neural network architecture 100 provided in the present invention identifies objects and key points in the detection head by making one judgment instead of two judgments. Next, the process enters optional step 940.

[0084] Optional step 940: Output the positions of the one or more object bounding boxes and the one or more key points. In this step, the positions of the key points may be the relative positions obtained in step 930, or the positions of the key points relative to the image calculated by the corresponding object bounding boxes.

[0085] An embodiment of the present invention provides a method for detecting objects and key points, comprising: receiving an image; executing a deep neural network architecture based on the image to obtain one or more object bounding boxes; executing a deep neural network architecture based on the one or more object bounding boxes to obtain relative positions of one or more key points corresponding to the one or more object bounding boxes; and outputting the positions of the one or more object bounding boxes and the one or more key points.

[0086] Preferably, in order to provide an image with more details under low light conditions, the image contains information in the infrared band.

[0087] Preferably, in order to utilize a neural network to implement a deep neural network architecture, the deep neural network architecture includes: a backbone architecture network; a neck network, which includes a feature pyramid network and a pyramid attention network to extract features from the backbone architecture network; and a detection head, which obtains the positions of one or more object bounding boxes and one or more key points from the neck network.

[0088] Preferably, in order to identify objects and key points in shallow and deep information, the detection head includes: a large object detection head, a medium object detection head and a small object detection head, which are used to obtain the positions of one or more object bounding boxes and one or more key points from multiple blocks of different sizes in the neck network respectively.

[0089] Preferably, in order to determine whether the object bounding box is credible when executing the deep neural network architecture, the deep neural network architecture is trained based on multiple candidate samples and multiple reference real samples. The threshold used to determine whether the predicted box is an object bounding box corresponds to the average and squared difference of the multiple candidate samples and their corresponding reference real samples.

[0090] Preferably, in order to train the object detection performance of the deep neural network architecture, the object box regression loss function of the deep neural network structure is related to the following two: the above-mentioned threshold; and the ratio of multiple intersections to unions between multiple candidate samples and their corresponding multiple benchmark real samples.

[0091] Preferably, in order to train the object key point detection performance of the deep neural network architecture, the key point loss function of the deep neural network architecture is a wing loss function.

[0092] Preferably, in order to make the deep neural network architecture have better key point detection performance, during the training process of the deep neural network architecture, one or any combination of the following algorithms is used to evaluate the detection performance of object key points during training: object key point similarity (OKS, Object Key-point Similarity) algorithm; and correct key point percentage (PCK, Percentage of Correct Key-points) algorithm.

[0093] Preferably, in order to enable the deep neural network architecture to provide the driver status within a vehicle (such as a car), one or more object bounding boxes and their corresponding one or more key point systems correspond to one or any combination of the following categories of objects: human face, human hand, mobile phone, cigarette, glasses and seat belt.

[0094] An embodiment of the present invention provides a host for object and key point detection, including: one or more processors, configured to execute a plurality of computer instructions stored in a non-volatile memory to implement the above-mentioned object and key point detection method.

[0095] One embodiment of the present invention provides a driver monitoring system for object and key point detection, including a host disposed in a vehicle (such as a car); and a photographic device for providing images.

[0096] In summary, the object and key point detection method provided by the present invention, as well as the host and driver monitoring system thereof, can realize the dual functions of object detection and key point detection through one-step judgment, reduce post-processing time, solve the problems caused by environmental and light changes, realize less processing and calculation requirements, reduce calculation time, and improve the real-time performance of the driver monitoring system.

Claims

1. A method for detecting an object and a key point, comprising: receiving an image; executing a deep neural network architecture based on the image to obtain one or more object bounding boxes; Executing the deep neural network architecture according to the one or more object bounding boxes to obtain relative positions of one or more key points corresponding to the one or more object bounding boxes; as well as The one or more object bounding boxes and the positions of the one or more key points are output.

2. The method for detecting objects and key points according to claim 1, characterized in that: The image contains information in the infrared band.

3. The object and key point detection method according to claim 1, characterized in that: The deep neural network architecture includes: Backbone architecture network; A neck network, comprising a feature pyramid network and a pyramid attention network to extract features from the backbone architecture network; and A detection head is provided to obtain one or more object bounding boxes and positions of one or more key points from the neck network.

4. The method for detecting objects and key points according to claim 3, characterized in that: The detection head includes: a large object detection head, a medium object detection head and a small object detection head, which are used to obtain one or more object bounding boxes and one or more key point positions from multiple blocks of different sizes of the neck network respectively.

5. The object and key point detection method according to claim 1, characterized in that: The deep neural network architecture is trained based on multiple candidate samples and multiple benchmark real samples, and the threshold for determining whether the predicted box is the object bounding box corresponds to the average value and square difference of the multiple candidate samples and their corresponding multiple benchmark real samples.

6. The method for detecting objects and key points according to claim 5, characterized in that: The object box regression loss function of the deep neural network architecture is related to the following two: the threshold value; and The ratio of the multiple intersections of the multiple candidate samples and their corresponding multiple reference real samples to the unions.

7. The object and key point detection method according to claim 1, characterized in that: The key point loss function of the deep neural network architecture is an airfoil loss function.

8. The method for detecting objects and key points according to claim 1, characterized in that: The deep neural network architecture uses one or any combination of the following algorithms to evaluate object keypoint detection performance during training: Object keypoint similarity algorithm; and Correct keypoint percentage algorithm.

9. The method for detecting objects and key points according to claim 1, characterized in that: The one or more object bounding boxes and their corresponding one or more key points correspond to one or any combination of the following categories of objects: human face, human hand, mobile phone, cigarette, glasses and seat belt.

10. A host for object and key point detection, comprising: One or more processors, configured to execute a plurality of computer instructions stored in a non-volatile memory, so as to implement the object and key point detection method according to any one of claims 1 to 9.

11. A driver monitoring system for object and key point detection, comprising: The host according to claim 10 is disposed in a carrier; as well as A photographic device is used to provide the image.

Images