Object detection device, object detection method, and storage medium

Abstract

This invention provides an object detection apparatus, an object detection method, and a storage medium that can improve the accuracy of object detection. The object detection apparatus includes: a high-dimensional vector transformation unit that transforms an input image into a high-dimensional vector; an object determination unit that determines whether a specified query in the input image is an object; an object region detection unit that detects regions of the object; a category detection unit that detects the type of the object; and an object detection unit that detects objects by processing the high-dimensional vector. The object detection unit detects objects based on the detection results of the object region detection unit and the category detection unit, as well as the determination result of the object determination unit. The detection result is based on a high-dimensional vector obtained through multiple processing steps of the high-dimensional vector, and the determination result is based on a high-dimensional vector obtained through prior processing steps.

Description

Technical Field

[0001] This invention relates to an object detection device, an object detection method, and a storage medium. Background Technology

[0002] In recent years, efforts to provide sustainable transportation systems that also take into account the vulnerable participants in transportation have become increasingly active. To achieve this goal, research and development efforts are being made to further improve the safety and convenience of transportation through technologies related to autonomous driving and safety precautions.

[0003] However, in autonomous driving and safety prevention technologies, research related to open-world object detection (OWOD), which can detect not only known objects but also unknown objects from images, is progressing. For example, Orr Zohar, Kuan-Chieh Wang, and Serena Yeung. Prob: Probabilistic objectness for open world object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 11444-11453, 2023. describes a technique for probabilistically detecting whether an object is an object without using pseudo-labels for unknown objects. The paper "Efficient detr: improving end-to-end object detector with dense prior" by Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang (arXivpreprint arXiv:2104.01318, 2021) describes a technique for object detection that uses a convolutional neural network (CNN) to transform an image into a feature map and applies a transformer to the feature map.

[0004] More specifically, the technology described in Non-Patent Document 1 is a technique that enables the decoder of a transformer to simultaneously learn inferences about objectivity (whether it is an object) and category. The technology described in Non-Patent Document 2 is a technique that inputs object queries for object detection and category classification into the decoder of a transformer and learns the object queries during the learning process.

[0005] However, in the technology described in Non-Patent Document 1, a conflict arises between the learning of object inference and the learning of category inference. Moreover, in the technology described in Non-Patent Document 2, object querying is not constituted as capturing the features of the object in the initial layer of the decoder, resulting in room for improvement in the accuracy of object detection in these prior art. Summary of the Invention

[0006] This invention was made in consideration of this situation, and one of its objectives is to provide an object detection apparatus, object detection method, and storage medium that can prevent conflicts between object-based inference learning and category inference learning in object detection that considers both known and unknown objects as objects, and capture object features from the initial layer of the decoder of the transformer, thereby improving the accuracy of object detection. This, in turn, contributes to the development of sustainable delivery systems.

[0007] The object detection device, object detection method, and storage medium of the present invention adopt the following structure.

[0008] (1): An object detection apparatus according to one aspect of the present invention comprises: a high-dimensional vector transformation unit that transforms an input image into a high-dimensional vector; an object determination unit that determines whether a query set in the input image is an object based on the high-dimensional vector; an object region detection unit that detects a region of the object based on the high-dimensional vector; a category detection unit that detects the type of the object based on the high-dimensional vector; and an object detection unit that detects objects by processing the high-dimensional vector, wherein the query includes a first query selected from a plurality of the high-dimensional vectors and an initialized second query, and the object detection unit processes the high-dimensional vector and detects the object based on the first query and the second query.

[0009] (2): In the above (1) scheme, the first query is a query selected in the manner of known objects distributed in the input image, and the second query is a query set in the manner of being distributed in the whole of the input image and learned in the manner of being distributed in the manner of known objects and unknown objects.

[0010] (3): In the above (1) scheme, the object detection unit detects the object based on the detection results of the object region detection unit and the category detection unit and the determination result of the object nature determination unit. The detection result is based on a high-dimensional vector obtained by processing the high-dimensional vector multiple times, and the determination result is based on a high-dimensional vector obtained by processing the high-dimensional vector before the multiple processing.

[0011] (4): In the above (1) scheme, the high-dimensional vector transformation unit, the object determination unit, the object region detection unit and the category detection unit are obtained by learning a deep learning model with a self-attention mechanism.

[0012] (5): In the above scheme (4), the deep learning model is a transformer model, the object detection unit detects the object based on the detection results of the object region detection unit and the category detection unit and the determination result of the object determination unit, the detection result is based on the high-dimensional vector obtained by processing the high-dimensional vector multiple times in each layer of the decoder of the transformer model, and the determination result is based on the high-dimensional vector obtained by processing the high-dimensional vector in each layer of the decoder before the multiple processing.

[0013] (6): In the above (1) scheme, the object determination unit calculates whether the query is an object as a first probability value, the category detection unit calculates the type of the query as a second probability value, and the object detection unit detects the object based on the first probability value and the second probability value.

[0014] (7): Another aspect of the present invention is an object detection method, wherein the object detection method causes a computer to perform the following processing: transforming an input image into a high-dimensional vector; determining, based on the high-dimensional vector, whether a query set in the input image is an object; detecting the region of the object based on the high-dimensional vector; detecting the type of the object based on the high-dimensional vector; and detecting the object by processing the high-dimensional vector, wherein the query includes a first query selected from a plurality of the high-dimensional vectors and an initialized second query, and the high-dimensional vector is processed and the object is detected based on the first query and the second query.

[0015] (8): Another aspect of the present invention stores a program in a storage medium, wherein the program causes a computer to perform the following processing: transforming an input image into a high-dimensional vector; determining, based on the high-dimensional vector, whether a query set in the input image is an object; detecting a region of the object based on the high-dimensional vector; detecting the type of the object based on the high-dimensional vector; and detecting the object by processing the high-dimensional vector, wherein the query includes a first query selected from a plurality of the high-dimensional vectors and an initialized second query, and the computer processes the high-dimensional vector and detects the object based on the first query and the second query.

[0016] According to the schemes in (1) to (8), in object detection where both known and unknown objects are considered as objects, it is possible to prevent conflicts between the learning of object-based inference and the learning of category inference, and to capture the features of objects from the initial layer of the transformer's decoder, thereby improving the accuracy of object detection. This, in turn, contributes to the development of sustainable delivery systems. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating an example of a system including an object detection device according to an embodiment of the present invention.

[0018] Figure 2 This is a diagram illustrating an example of the structure of teaching data.

[0019] Figure 3 This is a diagram illustrating an example of the structure of a machine learning model during the learning phase.

[0020] Figure 4 This is a diagram used to illustrate object queries generated by the query generation department.

[0021] Figure 5 This is a diagram used to illustrate the processing performed by the decoder unit.

[0022] Figure 6 This is a diagram illustrating an example of the structure of a machine learning model during the inference phase.

[0023] Figure 7 This is a diagram used to illustrate the effects of the method of the present invention.

[0024] Figure 8 This is a flowchart illustrating an example of the learning process of a machine learning model.

[0025] Figure 9 This is a flowchart illustrating an example of the inference process performed by a machine learning model. Detailed Implementation

[0026] Hereinafter, embodiments of the object detection apparatus, object detection method, and program of the present invention will be described with reference to the accompanying drawings.

[0027] [Overall Structure]

[0028] Figure 1 This is a diagram illustrating an example of a system 1 including an object detection device 100 according to an embodiment of the present invention. (See diagram below.) Figure 1 As shown, system 1 includes a vehicle M and an object detection device 100. Figure 1In this example, the vehicle M and the object detection device 100 are configured separately, but the object detection device 100 can also be mounted on the vehicle M. Furthermore, the cooperation between the object detection device 100 and the vehicle M is ultimately exemplary; the object detection device 100 can also be used independently without cooperation with the vehicle M. For example, the object detection device 100 can be installed as a web server or as an application on the user's terminal device (e.g., a personal computer or smartphone).

[0029] Vehicle M is, for example, a hybrid electric vehicle or an electric vehicle, and includes a camera to capture images of the exterior of vehicle M. While in motion, vehicle M transmits the images of the exterior captured by the camera to object detection device 100 via a network NW such as a cellular network, Wi-Fi network, or Internet.

[0030] The object detection device 100 receives an image sent from the vehicle M and detects objects reflected in the received image. The object detection device 100 includes, for example, an image acquisition unit 110, an image input unit 120, and a storage unit 130. The image acquisition unit 110, image input unit 120, and other components are implemented by executing a program (software) using a hardware processor such as a CPU (Central Processing Unit). Some or all of these components can be implemented using hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through a combination of software and hardware. The program can be pre-saved in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transitory storage medium), or stored in a removable storage medium such as a DVD or CD-ROM (a non-transitory storage medium), and installed by mounting the storage medium onto a drive device.

[0031] Storage unit 130 may be, for example, an HDD, flash memory, RAM (Random Access Memory), etc. Storage unit 130 may store teaching data 140 and machine learning model 150. It should be noted that, for ease of explanation, storage unit 130 stores teaching data 140, but the present invention is not limited to that structure, and may also store only the machine learning model 150 learned based on teaching data 140, as described later.

[0032] The image acquisition unit 110 acquires images from the vehicle M via the network NW. The image input unit 120 inputs the acquired images into the learned machine learning model 150 to detect objects reflected in the acquired images. The object detection device 100 may also send information related to the detected objects to the vehicle M via the network NW. Subsequently, based on the information related to the detected objects, the vehicle M may provide driving assistance to the occupants of the vehicle M (e.g., displaying the detected objects on the vehicle M's navigation device or outputting audible alarm information), or it may perform autonomous driving (e.g., generating a target trajectory to avoid the detected objects). The object detection device 100 may also combine this driving assistance function and autonomous driving function.

[0033] The machine learning model 150 includes, for example, a high-dimensional vector transformation unit 151, a query generation unit 152, a decoder unit 153, an object determination unit 154, an object region detection unit 155, a category detection unit 156, and an object detection unit 157 as its functional units. These functional units are also implemented, for example, by executing a program using a hardware processor such as a CPU. In this embodiment, the machine learning model 150 is a machine learning model obtained by adding two improvements to an object detection method using a transformer model, more specifically DETR (End-to-End Object Detection with Transformers), as detailed below.

[0034] [Teaching Data]

[0035] Figure 2 This is a diagram illustrating an example of the structure of teaching data 140. Teaching data 140 is a dataset consisting of multiple images IM that include both known objects and unknown objects that are the subjects of detection. For example, in Figure 2 In the image IM, there are known objects O1 as road signs, known objects O2 as people, and unknown objects O3 as plastic bags. In this embodiment, for known objects, the category label c representing the object, the center coordinates (x, y) of the bounding box surrounding the object (hereinafter, sometimes simply referred to as BB), the width w, and the height h are pre-associated. Alternatively, the bounding box can also be represented by a combination of the minimum and maximum x_min and y_max values ​​in the x-coordinate and the minimum and maximum y_min and y_max values ​​in the y-coordinate, so for known objects, (x_min, x_max, y_min, y_max) can also be associated.

[0036] Therefore, in Figure 2In the case of image IM, for a known object O1, the category label c1 representing a road sign, the center coordinates (x1, y1) of the BB, the width w1, and the height h1 are associated; for a known object O2, the category label c2 representing a person, the center coordinates (x2, y2) of the BB, the width w2, and the height h2 are associated. These image IMs, combined with the category label c, the center coordinates (x, y), the width w, and the height h (hereinafter, instance label L) associated with the known objects contained in the image IM, constitute teaching data 140.

[0037] [Learning Machine Learning Models]

[0038] Figure 3 This is a diagram illustrating an example of the structure of a machine learning model 150 during the learning phase. See below for reference. Figure 3 The learning process of machine learning model 150 will be explained. First, as... Figure 3 As shown in the upper left part, the high-dimensional vector transformation unit 151 inputs the image IM contained in the teaching data 140 into the backbone network. The backbone network includes a convolutional neural network (CNN), which transforms the input image IM into a d-dimensional feature map (feature quantity) with reduced dimension by performing CNN-based convolution on the CNN.

[0039] Next, the high-dimensional vector transformation unit 151 performs positional encoding on the feature maps output from the backbone network. Here, positional encoding refers to the process used to associate the feature maps with the positions of the original input image IM. By performing positional encoding, the object query described later can determine the position on the input image IM based on the feature maps.

[0040] Next, the high-dimensional vector transformation unit 151 inputs the feature maps with positional symbolization to the transformer encoder. Here, the transformer encoder outputs multiple queries (hereinafter sometimes referred to as intermediate object features) based on the feature maps output from the CNN, using a self-attention mechanism and a feed-forward layer. Here, the query represents a vector (high-dimensional vector) indicating which part of the image IM (which part is the object) should be considered. The processing in the machine learning model 150 from the input to the backbone network to the output by the transformer encoder is performed by the high-dimensional vector transformation unit 151.

[0041] Next, the query generation unit 152 performs query selection on the query output by the transformer encoder. Here, query selection refers to the process of selecting queries that have higher accuracy in representing objects in the image IM. As described above, the teaching data 140 includes instance labels L for known objects. Therefore, by learning query selection using the teaching data 140 associated with known objects, it is possible to select queries that focus on known objects in the image IM. More specifically, query selection is implemented by learning the parameters of a deep neural network (DNN) model that performs query selection based on the data of each known object. The learning of query selection is performed simultaneously with the learning of the object determination unit 154, the object region detection unit 155, and the category detection unit 156, which will be described later.

[0042] Furthermore, the query generation unit 152 generates an initialized learnable query. The learnable query is a query designed to extract unknown objects primarily from the image IM. The query generation unit 152 merges the query selected by the query selection with the initialized learnable query to generate an object query. That is, in the technology described in Non-Patent Document 2, the object query is not configured to capture object features in the initial layer of the decoder; however, based on this embodiment, the object query includes queries selected by the query selection that focus on known objects, and is therefore configured to capture object features from the initial layer of the decoder as described below.

[0043] The object query generated by the query generation unit 152 is input to the transformer decoder by the decoder unit 153. Here, the transformer decoder uses a self-attention mechanism, a cross-attention mechanism (multi-head attention), and a feedforward layer to multiply the intermediate object features with the object query to finally detect the object. For convenience, the number of object queries is N. N is a pre-defined hyperparameter.

[0044] Figure 4 This is a diagram used to illustrate the object query generated by the query generation unit 152. Figure 4 This represents the object query generated by the query generation unit 152 and input to the initial layer of the transformer decoder at that moment. Figure 4 In the table, the object query indicated by the black circle represents the query selected by the query selection, and the object query indicated by the black triangle represents the initialized learnable query.

[0045] like Figure 4As shown, the queries selected by the query selection process tend to be distributed (focused on) known objects (i.e., objects O1 and O2) in the image IM at the initial layer input to the transformer decoder. On the other hand, the initialized learnable queries are set to be distributed (focused on) the entire image IM regardless of whether the objects in the image IM are known or unknown. It should be noted that in Figure 4 In this invention, the initialized learnable queries are set at constant intervals on the image IM, but the invention is not limited to such a structure and can be initialized by any method such as random setting.

[0046] The decoder unit 153 inputs the object queries generated by the query generation unit 152 to the transformer decoder, which has multiple transformer decoder layers. The initial layer input of the transformer decoder consists of the intermediate object features output by the transformer encoder and the N object queries generated by the query generation unit 152, which are multiplied by a cross-attention mechanism, and outputs N object queries that more precisely capture the (known or unknown) objects in the image IM.

[0047] Similarly, the input to the subsequent layer of the transformer decoder consists of the intermediate object features output by the transformer encoder and the N object queries output by the previous layer of the transformer decoder, multiplied by a cross-attention mechanism, and outputs N object queries that more precisely capture the (known or unknown) objects in the image IM. This process is repeated until the final decoder unit 153 outputs N object queries from the final layer of the transformer decoder.

[0048] Figure 5 This is a diagram used to illustrate the process performed by the decoder unit 153. Figure 5 The left part indicates Figure 4 The same scene, Figure 5 The right-hand side represents the object query obtained after the processing performed by the decoder section 153 (i.e., the processing performed by each transformer decoder layer). For example... Figure 5 As shown, the initialized learnable query does not capture objects in the initial layer, but the processing performed by the decoder unit 153 results in the capture of both known objects (i.e., objects O1 and O2) and unknown objects (i.e., object O3). Thus, based on this embodiment, the object query includes the query selected by the query selection and the initialized learnable query, thereby enabling the capture of known objects from the initial layer, and the capture of both known and unknown objects in subsequent layers using the processing performed by the decoder unit 153.

[0049] The objectness determination unit 154 (Objectness Head) determines whether an object query is an object (objectness) based on the probability of each object query output by the transformer decoder layer (hereinafter, the output value output by the objectness determination unit 154 is sometimes referred to as p(o|q)). More specifically, in this embodiment, the probability value p(o|q) is modeled as a multivariate Gaussian distribution in the query embedding space, and the objectness determination unit 154 uses Mahalanobis distance and calculates the probability value p(o|q) = f using the following equation (1). obj (q).

[0050]

Mathematical Formula 1

[0051]

[0052] In equation (1), the symbol μ represents the mean of the multivariate Gaussian distribution, the symbol Σ represents the variance-covariance matrix of the multivariate Gaussian distribution, the symbol q represents the d-dimensional vector used as the query for each object, and the symbol d M This indicates the Mahalanobis distance in the query embedding space.

[0053] During the learning phase of the objectivity determination unit 154, in order to infer (optimize) the mean μ and variance-covariance matrix Σ of the multivariate Gaussian distribution, the sum of squared Mahalanobis distances shown by the following equation (2) is defined as the objectivity error function L. obj .

[0054]

Mathematical Formula 2

[0055]

[0056] In equation (2), the symbol Z represents the query q that matches a known object in the teaching data 140. i The set. By reducing the error function L obj The learning process involves optimizing the likelihood to learn the object-specific representation of each object query, p(o|q) = f. obj (q).

[0057] The object region detection unit 155 (BBox Head) detects the location of the object on the image IM as a bounding box for each object query output by the transformer decoder layer. More specifically, the object region detection unit 155 outputs a deep neural network (DNN) that generates the location information (center coordinates, width, height) of the bounding box corresponding to each object query. The object region detection unit 155 reduces the L1 loss and gIoU (generalized Intersection over Union) loss used for bounding box learning. bb And thus obtained.

[0058] The class detection unit 156 (Class Head) probabilistically detects the class of an object, assuming that each object query output by the transformer decoder layer is an object (hereafter, the output value output by the class detection unit 156 is sometimes referred to as p(c|o,q)). More specifically, the class detection unit 156 is a DNN that outputs the class (category) of an object as a probability value, assuming that each object query is an object. The class detection unit 156 reduces the sigmoid focal loss L used for class detection learning. cls The category detection unit 156 learns to output categories for known objects, and learns to output unknown categories for unknown objects.

[0059] Thus, each transformer decoder layer outputs N object queries. The object determination unit 154, the object region detection unit 155, and the category detection unit 156 take each object query q as input and output the probability value p(o|q), the bounding box position information, and the probability value p(c|o,q). During the learning phase of the machine learning model 150, the error function L of the object determination unit 154 is used. obj Error function L of object area detection unit 155 bb And the error function L of the category detection unit 156 cls The learning is carried out in a way that ultimately reduces the error function L represented by the following equation (3).

[0060]

Mathematical Expression 3

[0061]

[0062] As shown in equation (3), the error function L is composed of each error function L obj Error function L bb Error function L cls The summation is represented by the fact that, during the learning phase, object identification, bounding box identification, and category identification are learned simultaneously. However, conflicts sometimes arise between the learning of object identification and category identification. This is due to the following tendency: the learning of object identification moves towards separating and integrating the object's features from the background (thereby identifying a single object), while the learning of category identification moves towards separating and separating the object's features from each other (thereby identifying different objects).

[0063] Against this backdrop, in this embodiment, in the transformer decoder layer, object-orientation learning is stopped in the initial stage (e.g., the second layer), and only bounding box and category learning are performed in subsequent layers. For example, in Figure 3 In the case where the number of layers in the converter decoder is 6, but for layers from the first layer to the (n-1)th (2 < n < 6)th layer and the sixth layer, learning is performed in a way that reduces the error function L represented by equation (3). On the other hand, in layers n to 5, learning is performed in a way that reduces the error function L' = L. bb +L cls The learning process is performed in this manner. This suppresses the conflict between the learning of object-specific determination and the learning of category determination, and enables the object-specific determination unit 154, the object region detection unit 155, and the category detection unit 156 to learn. It should be noted that... Figure 3 The layer number 6 of the transformer decoder in the example is just one example; any positive integer layer number can be set.

[0064] [Inferences drawn from machine learning models]

[0065] Figure 6 This is a diagram illustrating an example of the structure of a machine learning model 150 in the inference phase. Figure 6 The machine learning model 150 shown is in Figure 5 The machine learning model 150 shown is the completed learning model obtained after completing the learning phase. Figure 5 The initialized learnable queries in Figure 6 The query is then learned. In the inference phase, the machine learning model 150 receives the image of the inferred object as input and sequentially performs processing by the high-dimensional vector transformation unit 151, the query generation unit 152, and the decoder unit 153.

[0066] As mentioned above, the learning of object-orientation determination stops at the initial layer stage in the transformer decoder layer. Therefore, in Figure 6 In the case of object determination, within the machine learning model 150 during the inference phase, the object determination unit 154 accepts the object query q from the nth layer as input and outputs the object probability value p(o|q). Conversely, the object region detection unit 155 accepts the object query q from the sixth layer as input and outputs the bounding box position information. Similarly, the category detection unit 156 accepts the object query q from the sixth layer as input and outputs the category probability value p(c|o,q).

[0067] In the inference phase, the object detection unit 157 detects objects from the input image that are the objects to be inferred based on the probability value p(o|q) output by the object determination unit 154, the position information of the bounding box output by the object region detection unit 155, and the probability value p(c|o,q) output by the category detection unit 156.

[0068] More specifically, the object detection unit 157 first calculates the final category inference value p(c|q) = p(c|o,q) × p(o|q) for each object query q. Next, the object detection unit 157 filters object queries q whose final category inference value p(c|q) is above a threshold, and then detects the filtered object queries q as objects. Alternatively, the object detection unit 157 may also filter the final category inference value p(c|q) to a predetermined number of object queries q.

[0069] When an object is detected by the object detection unit 157, the object detection device 100 can also provide information related to the detected object to the vehicle M for driving support or autonomous driving. For example, the object detection device 100 can also query q for the object detected as an object, merge the category information and the position information of the bounding box, and display them overlaid on the input image, and output them, for example, to the navigation device of the vehicle M. In addition, the vehicle M with autonomous driving function can also generate a driving track in a way that avoids the object query q detected as an object.

[0070] [Effects of the method of the present invention]

[0071] Figure 7 This is a diagram used to illustrate the effects of the method of the present invention. Figure 7 In the table shown, the upper part represents the performance of object detection tasks TASK1 to TASK4 based on the technology described in Non-Patent Document 1, and the lower part represents the performance of the same tasks based on the method of the present invention. For example... Figure 7 As shown, in each object detection task TASK1 to TASK4, it was confirmed that the method of the present invention has improved performance compared with the technology described in Non-Patent Document 1, both in terms of the detection performance of known objects (mAP) and the detection performance of unknown objects (U-Recall).

[0072] [Processing flow]

[0073] The following is for reference Figure 8 as well as Figure 9 The process of learning and inference of the machine learning model 150 of the present invention will be described. Figure 8 This is a flowchart illustrating an example of the learning process of machine learning model 150. Figure 8The processing of the flowchart shown can be performed manually or automatically by a "learning device" that is a computer device equipped with the flowchart.

[0074] First, the image of the teaching data 400 is input into the backbone network of the machine learning model 150 (step S100). Next, the feature map output from the backbone network is input into the encoder (transformer encoder) (step S102). Next, query selection is performed on the query output from the encoder, and the initialized learnable query is set as an object query (step S104). Next, the set object query is input into the decoder (while setting m=0) (step S106).

[0075] Next, set m = m + 1 (step S108). Next, determine whether the m-th decoder layer is more than or equal to the n-th layer (e.g., n ≥ 3) (step S110). If the m-th decoder layer is determined to be less than the n-th layer, calculate the loss L = L based on the outputs from the object determination unit 154, the object region detection unit 155, and the category detection unit 156. obj +L bb +L cls (Step S112). On the other hand, if it is determined that the m-th decoder layer is a layer above the n-th layer, then it is determined whether the m-th decoder layer is the sixth layer (Step S114).

[0076] If the m-th decoder layer is determined not to be the sixth layer, the loss L' = L is calculated based on the outputs of the object region detection unit 155 and the category detection unit 156. bb +L cls (Step S116). On the other hand, if the m-th decoder layer is determined to be the sixth layer, the loss L = L is calculated based on the output of the object determination unit 154 of the n-th layer and the output of the object region detection unit 155 and the category detection unit 156 of the sixth layer. obj +L bb +L cls (Step S118). Next, the machine learning model 150 is trained in a manner that reduces the loss calculated in steps S112, S114, and S118 (Step S120). Thus, the processing of this flowchart ends.

[0077] Figure 9 This is a flowchart illustrating an example of the inference process performed by machine learning model 150. Figure 9 The machine learning model 150 in the middle is based on Figure 8 The flowchart shown illustrates the completed learning of the model, which is then used by the object detection device 100.

[0078] First, the object detection device 100 inputs the image of the inferred object into the machine learning model 150 and obtains the object query (step S200). Next, for each object query, the object detection device 100 obtains the probability value p(o|q) output by the object determination unit 154 of the nth layer (step S202). Next, for each object query, the object detection device 100 obtains the probability value p(c|o,q) output by the category detection unit 156 of the sixth layer (step S204).

[0079] Next, the object detection device 100 uses the object detection unit 157 to calculate the inferred value p(c|q) = p(c|o,q) × p(o|q) for the final category of each object query (step S206). Next, the object detection device 100 uses the object detection unit 157 to filter object queries whose inferred value p(c|q) is above a threshold (step S208). Next, the object detection device 100 detects the filtered object queries as objects and merges them with the bounding box information output by the object region detection unit 155 of the sixth layer, and outputs the result (step S210). Thus, the processing of this flowchart ends.

[0080] It should be noted that in the above embodiments, the components of the machine learning model 150 include a high-dimensional vector transformation unit 151, a query generation unit 152, a decoder unit 153, an object determination unit 154, an object region detection unit 155, a category detection unit 156, and an object detection unit 157. However, these distinctions are for the sake of explanation. The present invention is at least based on the DETR technique, stopping the learning of object inference early and adding the initialized learnable query to the object query.

[0081] According to the embodiment described above, in object detection that targets both known and unknown objects, object inference learning is stopped at the initial layer of the transformer decoder. In subsequent layers, only category inference learning and object region detection learning are performed. The object query includes queries selected by query selection to capture known objects and learnable queries initialized to capture both known and unknown objects. That is, this prevents conflicts between object inference learning and category inference learning, and captures object features from the initial layer of the transformer decoder, thereby improving the accuracy of object detection.

[0082] The implementation methods described above can be performed as follows.

[0083] A driving assistance device comprising:

[0084] A storage medium that stores computer-readable instructions; and

[0085] The processor connected to the storage medium,

[0086] The processor is configured to perform the following processing by executing computer-readable instructions:

[0087] Transform the input image into a high-dimensional vector;

[0088] Based on the high-dimensional vector, determine whether the query set in the input image is an object;

[0089] Based on the high-dimensional vector, detect the region of the object;

[0090] Based on the high-dimensional vector, the type of the object is detected; and

[0091] Object detection is performed by processing the high-dimensional vector.

[0092] The query includes a first query selected from the plurality of high-dimensional vectors and an initialized second query.

[0093] The driving support device processes the high-dimensional vector and detects the object based on the first query and the second query.

[0094] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An object detection device, wherein, The object detection device includes: The high-dimensional vector transformation unit transforms the input image into a high-dimensional vector. The object determination unit determines, based on the high-dimensional vector, whether the query set in the input image is an object; An object region detection unit detects regions of the object based on the high-dimensional vector; The category detection unit detects the type of the object based on the high-dimensional vector; as well as The object detection unit detects objects by processing the high-dimensional vector. The query includes a first query selected from the plurality of high-dimensional vectors and an initialized second query. The object detection unit processes the high-dimensional vector and detects the object based on the first query and the second query.

2. The object detection device according to claim 1, wherein, The first query is a query selected in a manner that includes known objects distributed in the input image, and the second query is a query set in a manner that includes the entire input image and learned in a manner that includes both known and unknown objects.

3. The object detection device according to claim 1, wherein, The object detection unit detects the object based on the detection results of the object region detection unit and the category detection unit, as well as the determination result of the object nature determination unit. The detection result is based on a high-dimensional vector obtained by processing the high-dimensional vector multiple times, and the determination result is based on a high-dimensional vector obtained by processing the high-dimensional vector before the multiple processing.

4. The object detection device according to claim 1, wherein, The high-dimensional vector transformation unit, the object determination unit, the object region detection unit, and the category detection unit are obtained by learning through a deep learning model with a self-attention mechanism.

5. The object detection device according to claim 4, wherein, The deep learning model is a transformer model. The object detection unit detects the object based on the detection results of the object region detection unit and the category detection unit, as well as the determination result of the object nature determination unit. The detection result is based on a high-dimensional vector obtained by processing the high-dimensional vector multiple times in each layer of the decoder of the transformer model. The determination result is based on a high-dimensional vector obtained by processing the high-dimensional vector in each layer of the decoder before the multiple processing.

6. The object detection device according to claim 1, wherein, The object determination unit calculates whether the query is an object as a first probability value. The category detection unit calculates the category of the query as a second probability value. The object detection unit detects the object based on the first probability value and the second probability value.

7. An object detection method, wherein, The object detection method causes the computer to perform the following processing: Transform the input image into a high-dimensional vector; Based on the high-dimensional vector, determine whether the query set in the input image is an object; Based on the high-dimensional vector, detect the region of the object; Based on the high-dimensional vector, the type of the object is detected; as well as Object detection is performed by processing the high-dimensional vector. The query includes a first query selected from the plurality of high-dimensional vectors and an initialized second query. Based on the first query and the second query, the high-dimensional vector is processed and the object is detected.

8. A storage medium having a stored program, wherein, The program causes the computer to perform the following processing: Transform the input image into a high-dimensional vector; Based on the high-dimensional vector, determine whether the query set in the input image is an object; Based on the high-dimensional vector, detect the region of the object; Based on the high-dimensional vector, the type of the object is detected; as well as Object detection is performed by processing the high-dimensional vector. The query includes a first query selected from the plurality of high-dimensional vectors and an initialized second query. Based on the first query and the second query, the high-dimensional vector is processed and the object is detected.

Images