Object feature point detection device
By using detection models and parameter optimization techniques in the object feature point detection device, the position of the object feature point is directly inferred, and the problem of large computing load in the prior art is solved, and high-precision and efficient object detection are achieved.
Patent Information
- Application Number
- CN202380021361.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-02-14
- Publication Date
- 2024-11-08
AI Technical Summary
The existing object joint detection technology is difficult to achieve high-precision position inference and connection inference when the image segmentation is not detailed, resulting in heavy operation load.
The object feature point detection device with a detection model, a learning unit and a parameter optimization unit is adopted to optimize the detection model parameters by learning the correct answer data and inferred data in the image, and directly infer the position of the object feature point to reduce the image Area segmentation and traversal search to achieve efficient learning and detection.
It greatly reduces the computing load, realizes high-precision object feature point detection, reduces the calculation load of learning processing, and enables inexpensive computing devices to effectively detect object position.
Smart Images

Figure CN118922858A_ABST
Abstract
Description
Technical Field
[0001] An embodiment of the present invention relates to an object feature point detection device. Background Art
[0002] In recent years, various object joint detection technologies have been proposed for detecting joints of objects, such as human beings, and their positions based on camera images. In existing object joint detection technologies, camera images are segmented and object joint detection processing is performed for each region. Therefore, it is difficult to achieve high accuracy in position estimation and connection estimation without detailed image segmentation. As a result, the computational load in object joint detection is usually heavy.
[0003] In object feature point detection technologies such as object joint detection, any reduction in computational load is beneficial.
[0004] Patent Document 1: Japanese Patent Application Publication No. 2021-86322 Summary of the invention
[0005] Therefore, one of the objects of the present invention is to provide an object feature point detection device that can reduce the calculation load compared with the related art.
[0006] As an example, an object feature point detection device according to an embodiment of the present invention includes a detection model, a learning unit, and a parameter optimization unit. The detection model outputs inference data including the inferred position of each feature point contained in each of a plurality of objects in an input image. The learning unit executes the detection model and machine learning of the detection model. The parameter optimization unit optimizes parameters used to infer the position of each feature point of any object among the plurality of inference data output from the detection model that inputs a learning image obtained by photographing a plurality of objects. The detection model outputs inference data including the inferred position of each feature point contained in each of a plurality of objects in a newly input image using the parameters optimized by the parameter optimization unit.
[0007] According to the above structure, as an example, the type / position of the object and each feature point can be directly inferred without segmenting the image and searching the image over the entire area. Therefore, only the minimum required operation for detecting a predetermined number of multiple objects from the image is performed, which can reduce the operation load and achieve high-precision position detection.
[0008] In the object feature point detection device, the learning unit comprises: a learning data storage unit which stores a learning image obtained by photographing a plurality of objects and the correct answer data of each object in association with the correct answer position of each feature point contained in the objects in the learning image; and a calculation unit which uses the inference data of any object among the objects of the plurality of inference data output from the detection model to which the learning image is input, and the correct answer data of each of the objects, to calculate the sum of errors between the inference position and the correct answer position of each feature point of the arbitrary object, i.e., the total error, on a per-object basis, and associates the object corresponding to the minimum total error among the total errors of each of the objects with the arbitrary object, and determines the minimum total error of each object corresponding to each of the plurality of correct answer data and any one of the plurality of inference data as a total error group to be adopted for the optimization processing of the parameters of the detection model.
[0009] In addition, the correlation between the estimated positions of the feature points of the detection object, i.e., the object, and the correct answer positions of the feature points predefined for each object is performed in an arbitrary order. As a result, it is possible to achieve efficient learning of the detection model for detecting a predetermined number of multiple objects from an image without relying on the order of the correct answers.
[0010] In the object feature point detection device, the calculation unit may remove the object of the correct answer data associated with the object of the inference data from the association of the object of the correct answer data with respect to the objects of other inference data.
[0011] According to the above configuration, in the determination using the sum error group, when the number of detected objects for learning is set to N, it is sufficient to perform the sum error calculation processing of the sum of integers 1 to N. As a result, the calculation load in the learning process can be greatly reduced compared with the past, and the calculation for detecting the position of the object by an inexpensive calculation processing device becomes possible.
[0012] In the above-mentioned object feature point detection device, the above-mentioned calculation unit may also use the sum of the total errors contained in the above-mentioned total error group, that is, the loss function, to calculate the error, and the above-mentioned learning unit may also have an updating unit that updates the above-mentioned parameters of the above-mentioned detection model based on the above-mentioned total error using the above-mentioned loss function in the above-mentioned optimization processing.
[0013] According to the above configuration, it is possible to execute optimization processing of parameters taking into account the correct answer positions of the plurality of objects in the learning image, and thus it is possible to improve the convergence of learning.
[0014] In the above-mentioned object feature point detection device, the above-mentioned object may also be a person, and the above-mentioned feature points may also be joints of the human body.
[0015] According to the above configuration, it is possible to reduce the calculation load and detect the postures of multiple people with high accuracy.
[0016] As an example, an object feature point detection device of an embodiment of the present invention is an object feature point detection device having a detection model that outputs inference data of the inferred position of each feature point contained in each of a plurality of objects in an input image, the detection model using the inference data of any object among the plurality of inference data output by the detection model having input a learning image obtained by photographing a plurality of objects, and the correct answer data of each object including the correct answer position of each feature point contained in the objects in the learning image, to calculate the sum of errors between the inferred position of each feature point of the arbitrary object, i.e., the total error, with the object corresponding to the minimum total error among the total errors of each of the objects, and to associate the object with the arbitrary object, and to use the total error group to optimize the parameters by making each object of the plurality of correct answer data correspond to any one of the plurality of inference data.
[0017] Therefore, as an example, the type / position of the object and each feature point can be directly inferred without searching the image over the entire area. Therefore, only the minimum required operation for detecting a predetermined number of multiple objects from the image is performed, which can reduce the operation load and achieve high-precision position detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is a plan view of the interior of a vehicle in which the object joint detection device according to the embodiment is mounted, as viewed from above.
[0019] Figure 2 It is a block diagram showing the structure of the control system according to the embodiment.
[0020] Figure 3 This is a block diagram showing the functional structure of the ECU.
[0021] Figure 4 This is a diagram for explaining the outline of the detection model.
[0022] Figure 5 FIG. 1 is a diagram showing an example of a captured image serving as input data to a detection model.
[0023] Figure 6 This is a diagram showing an overview of learning data used for learning a learning detection model.
[0024] Figure 7This is a diagram showing an overview of the error group determination process performed in the learning of the detection model.
[0025] Figure 8 This is a diagram showing an overview of the error group determination process performed in the learning of the detection model.
[0026] Fig. 9 This is a diagram showing an overview of the error group determination process performed in the learning of the detection model.
[0027] Fig.10 This is a diagram showing an overview of a learning process including an error group determination process and a parameter optimization process.
[0028] Fig.11 This is a flowchart showing an example of the procedure of a learning process executed by the ECU as the object joint detection device. DETAILED DESCRIPTION
[0029] Hereinafter, a form of an object feature point detection device for implementing the present application (hereinafter referred to as “embodiment”) will be described in detail with reference to the drawings.
[0030] The object feature point detection device of the embodiment outputs inferred data including the inferred position of each feature point contained in each of the plurality of objects in the input image. Here, "object" refers to a mobile body (vehicle, two-wheeled vehicle, person, animal, robot, mechanical arm, drone, etc.), a three-dimensional structure, etc.
[0031] In addition, for the sake of specific description, the following example takes the case where the object to be detected is a person (human body) and the feature points of the object are joint points of the human body, that is, the case where the object feature point detection device is an object joint detection device.
[0032] In addition, according to this embodiment, the object feature point detection device of the present application is not limited to an object joint detection device. In addition, in each of the following embodiments, the same reference numerals are attached to the same parts, and repeated descriptions are omitted.
[0033] [1. Vehicle structure]
[0034] Figure 1 This is a plan view of the interior of a vehicle cabin of a vehicle 1 equipped with the object joint detection device according to the embodiment, as viewed from above.
[0035] Figure 11 is an example of a vehicle 1 having a rated passenger capacity of 5. A plurality of seats 2 are provided in the vehicle 1. A driver's seat 2a and a passenger seat 2b are provided at the front side of the vehicle, and a plurality of rear seats 2c to 2e are provided at the rear side. The rear seat 2c is provided behind the driver's seat 2a, the rear seat 2d is provided behind the passenger seat 2b, and the rear seat 2e is provided between the rear seats 2c and 2d.
[0036] The camera 4 is provided at the front side of the vehicle interior. The camera 4 has a built-in camera element such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor), and transmits an image captured by the camera element to an ECU 10 (see FIG. 1 ). Figure 2 The camera 4 captures the front of the vehicle 1 at a predetermined frame rate, for example.
[0037] As the photographing device 4, for example, any type of camera such as a single-lens camera, a stereo camera, a visible light camera, an infrared camera, or a TOF distance image camera can be used. In addition, among the above cameras, infrared photography is effective in that the camera is not easily bleached even when the outside of the vehicle is bright, and passengers can be captured to some extent even when the inside of the vehicle is dark.
[0038] [2. Configuration of Control System 100]
[0039] The vehicle 1 is provided with a control system 100 including an object joint detection device. Figure 2 The structure of the control system 100 will be described. Figure 2 It is a block diagram showing the structure of the control system 100 according to the embodiment.
[0040] like Figure 2 As shown, the control system 100 includes an in-vehicle network 3, a camera 4, a plurality of airbag devices 5, an alarm device 6, and an ECU 10. The ECU 10 is an example of an object joint detection device. The object joint detection device may include the camera 4 in addition to the ECU 10.
[0041] A plurality of airbag devices 5 are provided corresponding to each of the plurality of seats 2. The airbag device 5 protects the passengers sitting in the seats 2 from the impact by deploying the airbags when the vehicle 1 collides, etc. The alarm device 6 is composed of, for example, a warning light, a speaker, etc., and warns the passengers by light and sound. In addition, the alarm device 6 may also include a communication unit, and send a predetermined alarm information to a mobile terminal such as a smart phone held by the passenger.
[0042] ECU10 includes, for example, a CPU (Central Processing Unit) 11, an SSD (Solid State Drive) 12, a ROM (Read Only Memory) 13, and a RAM (Random Access Memory) 14. CPU11 implements the function of an object joint detection device by executing a program installed and stored in a non-volatile storage device such as ROM13. RAM14 temporarily stores various data used in the calculation of CPU11. SSD12 is an erasable non-volatile storage device that can store data even when the power supply of ECU10 is cut off. CPU11, ROM13, RAM14, etc. can be integrated in the same package. ECU10 can also be a structure that uses other logic operation processors such as DSP (Digital Signal Processor), logic circuits, etc. instead of CPU11. HDD (Hard Disk Drive) can also be provided instead of SSD12, and SSD12 or HDDE can also be provided separately from CU10.
[0043] In addition to the function of the object joint detection device, the ECU 10 also realizes various control functions of the vehicle 1. For example, the ECU 10 can control the airbag device 5 and the alarm device 6 by sending a control signal via the in-vehicle network 3. In addition, the ECU 10 can perform the control of the brake system, the control of the steering system, etc. In addition, the ECU 10 can obtain the image of the interior of the vehicle captured by the camera 4 from the camera 4 via the output line.
[0044] [3. Functional Structure of ECU 10]
[0045] Next, refer to Figure 3 The functional structure of the ECU 10 will be described. Figure 3 : is a block diagram showing the functional structure of the ECU 10 .
[0046] like Figure 3 As shown, the ECU 10 includes a detection unit 30, a determination unit 40, an in-vehicle device control unit 50, and a learning unit 60. The functions of the various units of the ECU 10 are implemented by the CPU 11 executing a program stored in the ROM 13. The above configuration may also be implemented by a hardware circuit.
[0047] The detection unit 30 executes the object position detection process using a learned detection model 64 obtained through machine learning described later.
[0048] The learned detection model 64 inputs an image and outputs the estimated positions of a predetermined number of objects contained in the acquired image. Specifically, the learned detection model is an AI (Artificial Intelligence) such as a learned neural network, for example, a learned model (inference model) such as DNN (Deep Neural Network) and CNN (Convolutional Neural Network).
[0049] Figure 4 This is a diagram for explaining object position detection processing using the learned detection model 64 . Figure 5 1 is a diagram showing an example of a detection image LI serving as input data of the learned detection model 64 .
[0050] like Figure 4 As shown, the learned detection model 64 inputs an image I obtained by photographing multiple objects, and outputs inferred data of the inferred position of each joint contained in each of the multiple objects in the image I. That is, the learned detection model 64 directly infers the type / position of the object and each feature point without searching the image throughout the entire area. In the following, for the sake of specific explanation, it is assumed that the detection image LI as input data contains 10 objects. In addition, Figure 5 In the detection image LI shown, only person A, person B, and person C are shown as examples.
[0051] Here, the inference data output by the learned detection model 64 is the following information. That is, when the detection image LI is input to the learned detection model 64, the inference data output by the learned detection model 64 is as follows: Figure 4 , 5 As shown, the information includes the identification (ID) of each object and the estimated position of each joint of each object. For example, the estimated data of person A is the identification information (ID) of the object, the estimated position (coordinates) of each joint such as the head (X, Y), the neck (X, Y), the right shoulder (X, Y), and the left shoulder (X, Y). The detection unit 30 outputs the estimated position of each joint of each object detected to the determination unit 40.
[0052] The determination unit 40 determines what kind of object exists in what posture (posture) and at what position in front of the vehicle 1 based on the estimated position of each joint of each object detected by the detection unit 30 . The determination unit 40 outputs the determination result to the vehicle-mounted device control unit 50 .
[0053] The vehicle-mounted device control unit 50 controls various devices mounted on the vehicle 1. As an example, the vehicle-mounted device control unit 50 individually controls the brake, accelerator, airbag device, etc. (an example of vehicle-mounted devices) based on the determination result of the determination unit 40.
[0054] The learning unit 60 performs a learning process of the learning detection model 640, optimizes the network parameters (hereinafter, also referred to as "parameters") of the learning detection model 640, and generates a learning-completed detection model 64. The learning unit 60 includes a learning data storage unit 61, a parameter storage unit 62, a setting unit 63, a learning detection model 640, a calculation unit 65, and an updating unit 66.
[0055] The learning data storage unit 61 stores a plurality of learning data including learning images obtained by photographing a predetermined number of objects and correct answer data (teacher data) for each object defined in the learning images.
[0056] Figure 6 640. The learning image LI, which is the input data of the learning detection model 640, is an image obtained by photographing a plurality of objects. Therefore, the learning image LI is a 10-object image (in Figure 6 , an image of three persons is illustrated.
[0057] In addition, the correct answer data is defined as the ID (identification information such as "person A") indicating what each of the multiple objects is, and the correct answer position (X, Y) of each joint of each object on the learning image LI. Figure 6 As shown, the correct answer data of person A on the learning image LI defines the head (X, Y), neck (X, Y), right shoulder (X, Y), left shoulder (X, Y), etc. as positions corresponding to the correct answer positions of each joint of person A. The learning data includes the learning image LI including 10 objects as input data, and the correct answer data of each object on the learning image LI. The necessary amount of the learning data is prepared for machine learning of the learning detection model 640.
[0058] The parameter storage unit 62 stores the network parameters set in the learning detection model 640. The setting unit 63 sets the learning image and the parameters in the learning detection model 640. The calculation unit 65 performs the adopted error calculation processing and the adopted total error group determination processing described later. The updating unit 66 performs the parameter optimization processing described later. In addition, the calculation unit 65 is an example of a parameter optimization unit.
[0059] [4. Learning Process]
[0060] The learning process of the learning detection model 640 performed by the learning unit 60 includes a process of determining the sum error group and a parameter optimization process. Here, the process of determining the sum error group is a process of efficiently determining the loss function of the machine learning of the learning detection model 640 by associating the inference position with the correct answer position in an arbitrary order. The parameter optimization process is a process of minimizing the loss function defined using the sum error group and optimizing the network parameters of the learning detection model 640. Each process is described below.
[0061] Figure 7 , Figure 8 , Fig. 9 It is a diagram for explaining the processing of determining the total error group.
[0062] First, if Figure 7 As shown in FIG. 1 , a learning image LI is input to the learning detection model 640, and a plurality of inference data E1 to inference data E2 are output as inference data. 10 .
[0063] Select multiple estimated data E1 to E1. 10 For each joint included in the selected inference data E1, the error between the inference position and the inference position of the corresponding joint included in the correct answer data GT1 of person A is calculated and the error is taken as the total error ea T1 The sum of the errors of each joint is calculated. Similarly, the selected inference data E1 and the remaining correct answer data GT2 to correct answer data GT2 of the plurality of persons are calculated. 10 The sum of the errors ea T2 ~Total error ea T10 The sum error operation is processed.
[0064] The total error ea obtained by the above calculation T1 ~Total error ea T10 The smallest total error (minimum total error) in is determined as the adopted total error ET1 for learning. Figure 7 In the example shown, the case where the total error ea1 adopts the total error ET1 (minimum total error) is shown.
[0065] Next, if Figure 8 As shown, the remaining multiple persons' estimated data E2 to estimated data E 10 For each joint included in the selected inference data E2, the error between the inferred position and the inferred position of the corresponding joint included in the correct answer data GT2 of person B is calculated and used as the total error eb T1The sum of the errors of each joint is calculated. Similarly, the selected inference data E2 and the remaining plurality of correct answer data GT3 to correct answer data GT3 are calculated. 10 The sum of the errors eb T2 ~Total error eb T9 The sum error operation is processed.
[0066] The calculated total error eb T1 ~Total error eb T9 The minimum total error in is determined as the adopted total error ET2 for learning. Figure 7 In the example shown, the total error eb is shown T2 This is the case where the total error ET2 (minimum total error) is used.
[0067] Next, if Fig. 9 As shown, multiple people's estimated data E3 to estimated data E 10 For each joint included in the selected inference data E3, the error between the inferred position and the inferred position of the corresponding joint included in the correct answer data GT2 of person B is calculated and the error is calculated as the total error ec T1 The sum of the errors of each joint is calculated. Similarly, the selected inference data E2 and the remaining plurality of correct answer data GT4 to correct answer data GT4 are calculated. 10 The sum of the individual errors ec T1 ~Total error ec T8 The sum error operation is processed.
[0068] The calculated total error ec T1 ~Total error ec T8 The minimum total error in is determined as the adopted total error ET3 for learning object C. In addition, Fig. 9 In the example shown, the total error ec is shown T1 This is the case where the total error ET3 (minimum total error) is used.
[0069] Next, the same process is performed on the remaining objects to determine the total errors ET1 to ET2 used for learning of each object. 10 As a result, the calculation of the total error ea1 to the total error ea 10 The 10-times sum error operation is processed.
[0070] In this embodiment, the total errors ET1 to ET 10This is referred to as an “adopted sum error group.” The loss function L1 used in the learning process using the learning data including the learning image LI1 and the correct answer data associated with the learning image LI1 can be defined using the adopted sum error group.
[0071] For example, the loss function L1 can be expressed as the summed error ET1 to ET2 constituting the summed error group. 10 In addition, if necessary, the loss function L1 can also be set to use the total error ET1~ET 10 The weighted linear sum of .
[0072] In the above-described adopted total error group determination process, when the number of objects to be learned is 10 as described above, the total error calculation process required until the adopted total errors of all objects are determined is 10+9+8+7+6+5+4+3+2+1=55 times. In addition, for example, when the number of objects to be learned is N, the total error calculation process required until the adopted total errors of all objects are determined is N(N+1) / 2 times.
[0073] Fig.10 This is a diagram for explaining an overview of a learning process including a total error group determination process and a parameter optimization process.
[0074] like Fig.10 As shown, the calculation unit 65 performs the sum error group determination processing in units of learning images and determines the loss function in units of learning images. Fig.10 In the process, the total error group is used to determine the treatment EI n is with the learning image LI n Corresponding processing.
[0075] The updating unit 66 updates the network parameters sequentially using the learning data and the corresponding loss function, and performs parameter optimization processing. As a method for parameter optimization processing, a general method such as gradient descent method, stochastic gradient descent method, error back-estimation method, etc. can be adopted.
[0076] The learning unit 60 stores the optimized parameters in the parameter storage unit 62. The learning unit 60 outputs the optimized parameters to the detection unit 30.
[0077] [5. Specific Operation of ECU 10]
[0078] Fig.11 : is a flowchart showing an example of the procedure of the learning process executed by the ECU 10 .
[0079] like Fig.11 As shown, the setting unit 63 reads the learning data from the learning data storage unit 61 (step S101).
[0080] Next, the setting unit 63 reads the parameters from the parameter storage unit 62 (step S102 ), and sets the read parameters in the learning detection model 640 (step S103 ).
[0081] Next, the learning image of the learning data read in step S101 is input to the learning detection model 640 (step S104 ).
[0082] The calculation unit 65 obtains each piece of estimated data output from the learning detection model 640 (step S105 ).
[0083] The calculation unit 65 selects predetermined (arbitrary) estimated data from among the acquired plurality of estimated data (step S106 ).
[0084] The calculation unit 65 performs a total error calculation process between each of the plurality of correct answer data of the learning data read in step S101 and the selected inference data (step S107 ). As a result, the total error for each of the plurality of objects is calculated.
[0085] The calculation unit 65 determines the minimum total error among the plurality of total errors calculated in step S107 as the adopted total error of the object corresponding to the selected estimation data (step S108 ).
[0086] The calculation unit 65 determines whether there are objects for which the total error is not determined (step S109). If the calculation unit 65 determines that there are remaining objects (yes in step S109), it repeatedly performs the processing of steps S106 to S108. On the other hand, if the calculation unit 65 determines that there are no remaining objects (no in step S109), it determines whether the processing has been performed on all the learning images (step S110).
[0087] When it is determined that the calculation unit 65 has processed all the learning images, the updating unit 66 performs parameter optimization processing using the sum error set (loss function) for each image (step S111). On the other hand, when it is determined that the calculation unit 65 has not processed all the learning images, the processing of steps S104 to S109 is repeatedly performed on the remaining learning images.
[0088] The object joint detection device of the embodiment described above has a detection model 64, a learning unit 60, and a calculation unit 65 as a parameter optimization unit. The detection model 64 outputs inference data including the inferred position of each feature point contained in each of the multiple objects in the input image. The learning unit 60 performs the detection model 64 and the machine learning of the detection model 64. The calculation unit 65 optimizes the parameters for inferring the position of each feature point of any object among the multiple inference data output from the detection model 64 to which the learning image obtained by shooting multiple objects is input. The detection model 64 uses the parameters optimized by the calculation unit 65 to output the inference data including the inferred position of each feature point contained in each of the multiple objects in the newly input image.
[0089] Therefore, as an example, the type / position of the object and each feature point can be directly inferred without segmenting the image and searching the image over the entire area. Therefore, only the minimum required operation for detecting a predetermined number of multiple objects from the image is performed, which can reduce the operation load and achieve high-precision position detection.
[0090] In addition, the learning unit 60 has a learning data storage unit 61 and a calculation unit 65. The learning data storage unit 61 stores the learning images obtained by photographing a plurality of objects and the correct answer data of each object at the correct answer position of each feature point contained in each object in the learning images in association with each other. The calculation unit 65 uses the inference data of any object among the objects of the plurality of inference data output from the learning detection model 640 to which the learning images are input, and the correct answer data of each object, and calculates the sum of errors between the inference position and the correct answer position of each feature point of any object, i.e., the total error, in units of objects. The calculation unit 65 associates the object corresponding to the minimum total error among the total errors of each object with the arbitrary object, and determines the minimum total errors for each object of the plurality of correct answer data to correspond to any object in the plurality of inference data as the adopted total error group for the optimization processing of the parameters of the detection model.
[0091] Therefore, the correlation between the estimated positions of the feature points of the detection object, i.e., the object, and the correct answer positions of the feature points pre-defined for each object is performed in an arbitrary order. As a result, efficient learning can be achieved for the detection model for detecting a predetermined number of multiple objects from an image, which is independent of the order of the correct answers.
[0092] In addition, the calculation unit 65 removes the correct answer position associated with the inferred position from the association with the correct answer position with respect to other inferred positions. That is, the learning unit 60 assigns the correct answer position closest (with a small error) to any one of the multiple inferred positions output by the learning detection model 640 in the learning process of the learning detection model 640. In addition, the learning unit 60 assigns the correct answer position closest (with a small error) to any one of the remaining (without a correct answer position assigned) of the multiple inferred positions output by the learning detection model 640 in the learning process of the learning detection model 640.
[0093] Therefore, in the decision of using the error group, when the number of detected objects for learning is set to N, it is sufficient to perform error calculation processing of the sum of integers from 1 to N. As a result, the calculation load in the learning process can be greatly reduced compared with the past, and calculation for detecting the position of the object by an inexpensive calculation processing device becomes possible.
[0094] The calculation unit 65 calculates the error using the loss function which is the sum of the errors included in the adopted error group. In the parameter optimization process, the learning unit 60 updates the parameters of the learning detection model 640 based on the error using the loss function which is the sum of the errors included in the adopted error group.
[0095] Therefore, it is possible to execute optimization processing of parameters taking into account the correct answer positions of the plurality of objects in the learning image, and to improve convergence in learning.
[0096] [Modification 1]
[0097] In the above embodiment, the object to be detected is a person, the feature points are the joints of the human body, and the object feature point detection device 1 is used as an example to describe the object. In contrast, the object to be detected is not limited to a person, and can also be a robot, a robot arm, a vehicle, or other mobile objects, etc., and various object objects.
[0098] For example, when the object to be detected is a robot or a robot arm, the joints and manipulator parts of the robot or the like can be set as feature points. In addition, when the object to be detected is a vehicle, the headlights, taillights, doors, etc. can be set as feature points. In any case, in the learning image, correct answer data including the correct answer position of each feature point of the object is prepared, and the above-mentioned learning process is performed using the correct answer data, so that the same effect can be achieved.
[0099] [Modification 2]
[0100] In the above-mentioned embodiment, as an example, an example of applying the learned detection model 64 in the process of detecting objects and their positions contained in an image captured in front of the vehicle 1 is described. In contrast, it is not necessary to be particularly limited to this example. For example, the learned detection model 64 can also be applied in the process of detecting objects and their positions contained in images captured from the side, rear, etc. of the vehicle 1. In addition, for example, the learned detection model 64 can also be applied in the process of detecting the position of the head of a passenger contained in an image captured in the interior of the vehicle. That is, as long as the number of multiple objects contained in the image is determined in advance, the learned detection model 64 can be applied in the same way even if an image of any subject is captured, and the positions of multiple objects contained in the image can be detected.
[0101] [Variation 3]
[0102] When executing the above-mentioned error calculation processing, the calculation unit 65 may also increase the weight of the width W in the horizontal axis and the height H in the vertical axis included in each correct answer position and each inferred position compared with the coordinates (X, Y) included in each correct answer position and each inferred position. That is, the size and shape of the bounding box can be adjusted according to the detection object.
[0103] [Variation 4]
[0104] In the above embodiment, the case where the learning unit 60 that performs the learning process of the learning detection model 640 and the detection unit 30 having the learning detection model 640 that performs the inference process are built into the same device is exemplified. In contrast, the learning unit 60 and the detection unit 30 can also be configured as different devices. In addition, for example, the learning unit 60 can be implemented by a computer on the cloud.
[0105] Although the embodiments of the present invention are illustrated above, the above embodiments and modifications are merely examples and are not intended to limit the scope of the invention. The above embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, combinations, and changes can be made without departing from the scope of the invention. In addition, the structures and shapes of each embodiment and each modification can also be partially replaced for implementation.
[0106] Description of Reference Numerals
[0107] 1…vehicle, 2…seat, 3…in-vehicle network, 4…camera, 5…airbag device, 6…alarm device, 10…ECU, 30…detection unit, 40…judgment unit, 50…vehicle equipment control unit, 60…learning unit, 61…learning data storage unit, 62…parameter storage unit, 63…setting unit, 64…learning completion detection model, 65…calculation unit, 66…update unit, 100…control system, 640…learning detection model.
Claims
1. An object feature point detection device, comprising: a detection model, the output of which includes inferred data of the inferred position of each feature point contained in each of the plurality of objects in the input image; A learning unit, which performs machine learning of the above detection model; as well as a parameter optimization unit that optimizes a parameter for estimating a position of each feature point of an arbitrary object among the plurality of the inference data output from the detection model to which a learning image obtained by photographing a plurality of objects is input, The detection model outputs estimation data including the estimated position of each feature point included in each of the plurality of objects in the newly input image, using the parameters optimized by the parameter optimization unit.
2. The object feature point detection device according to claim 1, wherein: The above-mentioned learning department has: a learning data storage unit that stores a learning image obtained by photographing a plurality of objects and correct answer data for each object including a correct answer position of each feature point included in the learning image in association with each object; as well as A calculation unit, which uses the inference data of any object among the multiple inference data output from the detection model to which the learning image is input, and the correct answer data of each of the objects, to calculate the sum of errors between the inference position of each feature point of the arbitrary object and the correct answer position, that is, the total error, with the object as the unit, associates the object corresponding to the minimum total error among the total errors of each of the objects with the arbitrary object, and determines the minimum total error of each object corresponding to the multiple correct answer data and any object in the multiple inference data as a total error group to be adopted for optimization processing of parameters of the detection model.
3. The object feature point detection device according to claim 2, wherein: The calculation unit removes the object of the correct answer data associated with the object of the inference data from the association with the object of the correct answer data with respect to the objects of other inference data.
4. The object feature point detection device according to claim 2, wherein: The calculation unit calculates the error using the sum of the total errors included in the total error group, that is, the loss function. The learning unit further includes an updating unit that updates the parameters of the detection model based on the total error using the loss function in the optimization process.
5. The object feature point detection device according to claim 1 or 2, wherein: The object is a person, and the feature points are joints of the human body.
6. An object feature point detection device, comprising a detection model that outputs inferred data including inferred positions of each feature point contained in each of a plurality of objects in an input image, wherein: The above detection model Using the inference data of any object among the multiple inference data output from the above-mentioned detection model that has input a learning image obtained by photographing multiple objects, and the correct answer data of each object including the correct answer position of each feature point contained in the above-mentioned objects in the learning image, the sum of the errors between the above-mentioned inference position and the above-mentioned correct answer position of each feature point of the above-mentioned arbitrary object, that is, the total error, is calculated based on the above-mentioned object, and the object corresponding to the minimum total error among the above-mentioned total errors of each of the above-mentioned objects is associated with the above-mentioned arbitrary object, and the minimum total error used to make each object of the multiple correct answer data correspond to any object in the multiple inference data is used as the total error group to optimize the parameters.
