Image processing method, image processing device and recording medium

Abstract

An image processing device (100) detects the front end of an object from an image. The image processing device (100) includes an image input unit (110) that receives an input of an image, a feature map generation unit (112) that generates a feature map by applying a convolution operation to the image, a first conversion unit (114) that generates a first output by applying a first conversion to the feature map, a second conversion unit (116) that generates a second output by applying a second conversion to the feature map, and a third conversion unit (118) that generates a third output by applying a third conversion to the feature map. The first output indicates information related to a predetermined number of candidate regions on the image, the second output indicates the likelihood of whether the front end of the object exists in the candidate region, and the third output indicates information related to the direction of the front end of the object existing in the candidate region.

Description

Technical Field

[0001] The invention relates to an image processing method and an image processing device. Background Art

[0002] In recent years, deep learning, which is a neural network with deep network layers, has attracted much attention. For example, Patent Document 1 proposes a technique for applying deep learning to detection processing.

[0003] In the technology described in Patent Document 1, detection processing is achieved by learning whether each of a plurality of regions arranged at equal intervals on an image includes a detection object, and if so, how to move or deform the region to better fit the detection object.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun "FasterR-CNN: Towards Real-Time Object Detection with Region Proposal Networks", Conference on Neural Information Processing Systems (NIPS), 2015 Summary of the invention

[0007] Problem that the invention aims to solve

[0008] In the detection process of the tip of an object, in addition to its position, the direction may also be important. However, in the conventional technology described in Patent Document 1, the direction cannot be taken into consideration.

[0009] The present invention has been made in view of such circumstances, and an object of the present invention is to provide a technology capable of considering not only the position but also the direction of the tip of an object in the detection process of the tip of the object.

[0010] Means used to solve problems

[0011] In order to solve the above-mentioned problem, an image processing device according to a certain scheme of the present invention is used to detect the front end of an object from an image, and comprises: an image input unit, which accepts an input of an image; a feature map generating unit, which generates a feature map by applying a convolution operation to the image; a first conversion unit, which generates a first output by applying the first conversion to the feature map; a second conversion unit, which generates a second output by applying the second conversion to the feature map; and a third conversion unit, which generates a third output by applying the third conversion to the feature map, wherein the first output represents information related to a predetermined number of candidate regions on the image, the second output represents the likelihood of whether the front end of the object exists in the candidate region, and the third output represents information related to the direction of the front end of the object existing in the candidate region.

[0012] Another embodiment of the present invention is also an image processing device. The device is used to detect the front end of an object from an image, and comprises: an image input unit that receives an input of an image; a feature map generation unit that generates a feature map by applying a convolution operation to the image; a first conversion unit that generates a first output by applying the first conversion to the feature map; a second conversion unit that generates a second output by applying the second conversion to the feature map; and a third conversion unit that generates a third output by applying the third conversion to the feature map, wherein the first output represents information related to a predetermined number of candidate points on the image, the second output represents the likelihood of whether the front end of the object exists near the candidate point, and the third output represents information related to the direction of the front end of the object that exists near the candidate point.

[0013] Another embodiment of the present invention is an image processing method. The method is an image processing method for detecting the front end of an object from an image, comprising: an image input step for accepting an image input; a feature map generation step for generating a feature map by applying a convolution operation to the image; a first conversion step for generating a first output by applying the first conversion to the feature map; a second conversion step for generating a second output by applying the second conversion to the feature map; and a third conversion step for generating a third output by applying the third conversion to the feature map, wherein the first output represents information related to a predetermined number of candidate regions on the image, the second output represents the likelihood of whether the front end of the object exists in the candidate region, and the third output represents information related to the direction of the front end of the object existing in the candidate region.

[0014] Furthermore, arbitrary combinations of the above-described constituent elements and embodiments in which the present invention is expressed in the form of methods, apparatuses, systems, recording media, computer programs, and the like may also be practiced as additional embodiments of the present invention.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide a technique capable of considering not only the position but also the direction in the detection process of the tip of an object. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a block diagram showing the functional structure of the image processing device according to the embodiment.

[0018] Figure 2 It is used to illustrate that Figure 1 A diagram showing the effect of considering the reliability of the direction of the distal end of the treatment tool in determining whether the candidate region includes the distal end of the treatment tool by the candidate region determination unit.

[0019] Figure 3 This is a diagram for explaining the effect of considering the direction of the tip of the treatment instrument in determining the candidate region to be deleted. DETAILED DESCRIPTION

[0020] Hereinafter, the present invention will be described based on preferred embodiments with reference to the drawings.

[0021] Figure 1 1 is a block diagram showing the functional structure of the image processing device 100 of the embodiment. Each block shown here can be implemented by a component or mechanical device represented by a CPU (central processing unit) and a GPU (Graphics Processing Unit) of a computer in terms of hardware, and can be implemented by a computer program or the like in terms of software, but the functional blocks implemented by their cooperation are described here. Therefore, those skilled in the art who have access to this specification should understand that these functional blocks can be implemented in various forms through a combination of hardware and software.

[0022] The following description uses the example of using the image processing device 100 to detect the front end of an endoscope's treatment instrument, but those skilled in the art should understand that the image processing device 100 can also be used to detect the front ends of objects other than these, specifically, the front ends of other objects such as robotic arms, needles under a microscope, and rod-shaped props used in sports.

[0023] The image processing device 100 is a device for detecting the front end of an endoscope treatment instrument based on an endoscope image. The image processing device 100 includes an image input unit 110, a correct answer input unit 111, a feature map generation unit 112, a region setting unit 113, a first conversion unit 114, a second conversion unit 116, a third conversion unit 118, a comprehensive score calculation unit 120, a candidate region determination unit 122, a candidate region deletion unit 124, a weight initialization unit 126, an overall error calculation unit 128, an error propagation unit 130, a weight update unit 132, a result presentation unit 133, and a weight coefficient storage unit 134.

[0024] First, an application process of detecting the distal end of a treatment instrument from an endoscopic image by the learned image processing device 100 will be described.

[0025] The image input unit 110 receives an input of an endoscopic image, for example, from a video processor or other device connected to the endoscope. The feature map generation unit 112 applies a convolution operation using a prescribed weight coefficient to the endoscopic image received by the image input unit 110, thereby generating a feature map. The weight coefficient is obtained in a learning process described later and is stored in a weight coefficient storage unit 134. In this embodiment, a convolutional neural network (CNN: Convolutional Neural Network) based on VGG-16 is used as a convolution operation, but it is not limited to this, and other CNNs can also be used. For example, as a convolution operation, a Residual Network that has Identity Mapping (IM) introduced can also be used.

[0026] The region setting unit 113 sets a predetermined number of regions (hereinafter referred to as “initial regions”), for example, at equal intervals on the endoscopic image received by the image input unit 110 .

[0027] The first conversion unit 114 generates information (first output) related to multiple candidate areas corresponding to the multiple initial areas, respectively, by applying the first conversion to the feature map. In the present embodiment, the information related to the candidate area is information including the amount of position change used to bring the reference point (e.g., the center point) of the initial area closer to the front end. In addition, the information related to the candidate area is not limited to this, and for example, it may also be information including the position and size of the area after the initial area is moved in order to be more suitable for the front end of the treatment instrument. In the first conversion, a convolution operation using a specified weight coefficient is used. The weight coefficient is obtained in the learning process described later and is stored in the weight coefficient storage unit 134.

[0028] The second conversion unit 116 generates the likelihood (second output) of whether the front end of the treatment instrument exists in each of the plurality of initial regions by applying the second conversion to the feature map. In addition, the second conversion unit 116 may also generate the likelihood of whether the front end of the treatment instrument exists in each of the plurality of candidate regions. In the second conversion, a convolution operation using a predetermined weight coefficient is used. The weight coefficient is obtained in a learning process described later and is stored in the weight coefficient storage unit 134.

[0029] The third conversion unit 118 generates information (third output) related to the direction of the tip of the treatment instrument respectively existing in the plurality of initial regions by applying the third conversion to the feature map. Alternatively, the third conversion unit 118 may generate information related to the direction of the tip of the treatment instrument respectively existing in the plurality of candidate regions. In this embodiment, the information related to the direction of the tip of the treatment instrument is a direction vector (v ) extending from the tip of the treatment instrument as a starting point and along an extension line of the extension direction of the tip. x , v y In the third conversion, a convolution operation using a predetermined weight coefficient is used. The weight coefficient is obtained in a learning process described later and is stored in the weight coefficient storage unit 134.

[0030] The comprehensive score calculation unit 120 calculates the comprehensive score of each of the multiple initial regions or the comprehensive score of each of the multiple candidate regions based on the likelihood generated by the second conversion unit 116 and the reliability of the information related to the direction of the front end of the treatment instrument generated by the third conversion unit 118. In this embodiment, the "reliability" of the information related to the direction refers to the size of the direction vector of the front end. The comprehensive score calculation unit 120 calculates the comprehensive score (Score) based on the weighted sum of the likelihood and the reliability of the direction, specifically according to the following formula (1). total ).

[0031] [Formula 1]

[0032]

[0033] Here, Score2 is the likelihood, and w3 is the weight coefficient multiplied by the magnitude of the direction vector.

[0034] The candidate region determination unit 122 determines whether the tip of the treatment tool is included in each of the plurality of candidate regions based on the comprehensive score, and as a result, determines (presumes) the candidate region where the tip of the treatment tool is located. Specifically, the candidate region determination unit 122 determines that the tip of the treatment tool is present in a candidate region whose comprehensive score is greater than a predetermined threshold.

[0035] Figure 21 is a diagram for explaining the effect of using a comprehensive score in determining whether a candidate region includes the front end of a treatment instrument by a candidate region determination unit 122, that is, considering not only the likelihood but also the size of the direction vector of the front end of the treatment instrument in determining the candidate region. In this example, the treatment instrument 10 is bifurcated and has a protrusion 12 at a branch portion that branches into two branches. The protrusion 12 has a shape similar to the front end of the treatment instrument in part, and therefore, the likelihood of a candidate region 20 including the protrusion 12 is sometimes output as being high. In this case, when only the likelihood is used to determine whether the candidate region is where the front end 14 of the treatment instrument 10 is located, the candidate region 20 may be determined as the candidate region where the front end 14 of the treatment instrument 10 is located, that is, the protrusion 12 of the branch portion may be erroneously detected as the front end of the treatment instrument. In contrast, in the present embodiment, as described above, in addition to the likelihood, the size of the direction vector of the front end is also considered to determine whether the candidate region is where the front end 14 of the treatment instrument 10 is located. Since the magnitude of the direction vector of the protrusion 12 that is not the branch portion of the distal end 14 of the treatment instrument 10 tends to be small, the detection accuracy can be improved by considering the magnitude of the direction vector in addition to the likelihood.

[0036] return Figure 1 , the candidate region deletion unit 124 calculates the similarity between the plurality of candidate regions when the candidate region determination unit 122 determines that the front end of the treatment tool exists in the plurality of candidate regions. Then, when the similarity is above the predetermined threshold and the directions of the front ends of the treatment tools corresponding to the plurality of candidate regions are substantially consistent, they are considered to be candidate regions where the same front end is detected, and therefore, the candidate region deletion unit 124 retains the corresponding candidate regions with higher comprehensive scores and deletes the candidate regions with lower comprehensive scores. On the other hand, when the similarity is less than the predetermined threshold, or when the directions of the front ends of the treatment tools corresponding to the plurality of candidate regions are different from each other, they are considered to be candidate regions where different front ends are detected, and therefore, the candidate region deletion unit 124 does not delete any candidate region and retains all of them. In addition, regarding the case where the directions of the front ends of the treatment tools are substantially consistent, in addition to the case where the directions of the front ends of each other are parallel, it also refers to the case where the acute angle formed by the directions of the front ends of each other is below the predetermined threshold. In addition, in this embodiment, for the similarity, the degree of overlap (Intersection over Union) between the candidate regions is used. That is, the more the candidate regions overlap, the higher the similarity. In addition, the similarity is not limited to this, and for example, the inverse of the distance between candidate regions may be used.

[0037] Figure 3: is a diagram for explaining the effect of considering the direction of the front end in determining the candidate region to be deleted. In this example, the first candidate region 40 detects the front end of the first treatment instrument 30, and the second candidate region 42 detects the front end of the second treatment instrument 32. In the case where the front end of the first treatment instrument 30 is close to the front end of the second treatment instrument 32, and the first candidate region 40 is close to the second candidate region 42, when the deletion is determined only by their similarity, although the first candidate region 40 and the second candidate region 42 are candidate regions where the front ends of different treatment instruments are detected, it may be determined that one of the candidate regions is deleted. That is, it is considered that the first candidate region 40 and the second candidate region 42 detect the same front end and one of the candidate regions is deleted. In contrast, the candidate area deletion unit 124 of the present embodiment considers the direction of the front end in addition to the similarity to decide whether to delete the candidate area. Therefore, even if the first candidate area 40 and the second candidate area 42 are close and have a high similarity, since the direction D1 of the front end of the first treatment instrument 30 detected by them is different from the direction D2 of the front end of the second treatment instrument 32, neither candidate area is deleted. Therefore, the front end of the first treatment instrument 30 and the front end of the second treatment instrument 32 that are close to each other can be detected.

[0038] return Figure 1 The result presenting unit 133 presents the detection result of the tip of the treatment tool on a display, for example. The result presenting unit 133 presents the candidate region that is determined by the candidate region determining unit 122 to contain the tip of the treatment tool and is not deleted by the candidate region deleting unit 124, as the candidate region for detecting the tip of the treatment tool.

[0039] Next, a learning process of learning (optimizing) each weight coefficient used in each convolution operation of the image processing device 100 will be described.

[0040] The weight initialization unit 126 initializes each weight coefficient to be learned, and each weight coefficient is a weight coefficient used in each process of the feature map generation unit 112, the first conversion unit 114, the second conversion unit 116, and the third conversion unit 118. Specifically, the weight initialization unit 126 uses the average 0, the standard deviation wscale / √(c i ×k×k) normal random number. wscale is the scale parameter, c i is the number of input channels of the convolution layer, and k is the convolution kernel size. In addition, as the initial value of the weight coefficient, a weight coefficient learned from a large-scale image DB different from the endoscopic image DB used in this learning can also be used. Thus, even when the number of endoscopic images used for learning is small, the weight coefficient can be learned.

[0041] The image input unit 110 receives input of an endoscopic image for learning, for example, from a user terminal or other device. The correct answer input unit 111 receives correct answer data corresponding to the endoscopic image for learning from a user terminal or other device. In the correct answer corresponding to the output of the processing of the first conversion unit 114, the following position change amount is used, which is used to make the reference point (center point) of each of the multiple initial areas set by the area setting unit 113 on the endoscopic image for learning consistent with the front end of the treatment instrument, that is, the position change amount indicates how to move each of the multiple initial areas closer to the front end of the treatment instrument. In the correct answer corresponding to the output of the processing of the second conversion unit 116, a binary value indicating whether the front end of the treatment instrument exists in the initial area is used. In the correct answer corresponding to the third conversion, a unit direction vector indicating the direction of the front end of the treatment instrument existing in the initial area is used.

[0042] The processing performed by the feature map generation unit 112 , the first conversion unit 114 , the second conversion unit 116 , and the third conversion unit 118 in the learning process is the same as the processing in the application process.

[0043] The overall error calculation unit 128 calculates the overall error of the processing based on the outputs of the first conversion unit 114, the second conversion unit 116, and the third conversion unit 118 and the correct answer data corresponding thereto. The error propagation unit 130 calculates the error in the processing of the feature map generation unit 112, the first conversion unit 114, the second conversion unit 116, and the third conversion unit 118 based on the overall error.

[0044] The weight updating unit 132 updates the weight coefficients used in the convolution operations of the feature map generating unit 112, the first conversion unit 114, the second conversion unit 116, and the third conversion unit 118 based on the error calculated by the error propagation unit 130. In addition, in the method of updating the weight coefficient based on the error, for example, a probabilistic gradient descent method can also be used.

[0045] Next, the operation of the image processing apparatus 100 configured as described above during application will be described.

[0046] The image processing device 100 first sets a plurality of initial regions for the received endoscopic image. Next, the image processing device 100 applies a convolution operation to the endoscopic image to generate a feature map, applies a first operation to the feature map to generate information related to a plurality of candidate regions, applies a second operation to the feature map to generate the likelihood that the front end of the treatment instrument exists in each of the plurality of initial regions, and applies a third operation to the feature map to generate information related to the direction of the front end of the treatment instrument respectively existing in the plurality of initial regions. Then, the image processing device 100 calculates a comprehensive score for each candidate region, and identifies a candidate region having a comprehensive score above a predetermined threshold as a candidate region in which the front end of the treatment instrument is detected. In addition, the image processing device 100 calculates the similarity between the identified candidate regions, and based on the similarity, deletes the candidate region with a lower likelihood among the candidate regions in which the same front end is detected. Finally. The image processing device 100 prompts the candidate region that is not deleted and remains as a candidate region for detecting the front end of the treatment instrument.

[0047] According to the image processing apparatus 100 described above, information on the direction of the tip of the treatment tool is considered in determining the candidate region where the tip of the treatment tool is located, that is, detecting the tip of the treatment tool.

[0048] The present invention has been described above based on the embodiments. These embodiments are illustrative, and those skilled in the art will appreciate that various modifications can be made to the combinations of their respective components or processing steps, and such modifications are also within the scope of the present invention.

[0049] As a variation, the image processing device 100 may set a predetermined number of points (hereinafter referred to as "initial points") on the endoscopic image, for example, at equal intervals, and generate information related to multiple candidate points corresponding to the multiple initial points by applying a first transformation to the feature map (first output), generate a likelihood of whether the front end of the treatment device exists in the vicinity of each of the initial points or in the vicinity of each of the multiple candidate points (for example, within a specified range from each point) by applying a second transformation (second output), and generate information related to the direction of the front end of the treatment device that exists in the vicinity of each of the multiple initial points or in the vicinity of each of the multiple candidate points by applying a third transformation (third output).

[0050] In the embodiments and variations, the image processing device may also include storage devices such as processors and memories. Regarding the processor here, for example, the functions of each part can be implemented by independent hardware, or the functions of each part can also be implemented by integrated hardware. For example, the processor includes hardware, and the hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the processor can be composed of one or more circuit devices (such as ICs, etc.) mounted on a circuit substrate, and one or more circuit elements (such as resistors, capacitors, etc.). The processor may also be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as GPU (Graphics Processing Unit) or DSP (Digital Signal Processor) can be used. In addition. The processor may also be a hardware circuit based on ASIC (Application Specific Integrated Circuit) or FPGA (Field-programmable Gate Array). In addition, the processor may also include an amplifier circuit or a filter circuit for processing analog signals. The memory may be a semiconductor memory such as SRAM, DRAM, or a register, or a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. For example, the memory stores commands that can be read by a computer, and the processor executes the commands to realize the functions of various parts of the image processing device. The commands here may be commands that constitute a command set of a program, or commands that instruct the hardware circuit of the processor to act.

[0051] In the embodiment and the modified example, the processing units of the image processing apparatus may be connected via any form or medium of digital data communication such as a communication network. Examples of the communication network include LAN, WAN, and computers and networks forming the Internet.

[0052] Description of symbols

[0053] 100 image processing device, 110 image input unit, 112 feature map generation unit, 114 first conversion unit, 116 second conversion unit, 118 third conversion unit.

[0054] Industrial Applicability

[0055] The invention relates to an image processing method and an image processing device.

Claims

1. An image processing device for detecting the front end of an object from an image, characterized in that: The image processing device comprises: An image input unit that receives image input; a feature map generating unit, which generates a feature map by applying a convolution operation to the image; a first transformation unit for generating a first output by applying a first transformation to the feature map; a second transformation unit that generates a second output by applying a second transformation to the feature map; and a third transformation unit that generates a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate regions on the image, The second output indicates the likelihood of whether the front end of the object exists in the candidate area. The third output represents information related to the direction of the front end of the object existing in the candidate area, The first conversion unit, the second conversion unit and the third conversion unit respectively apply convolution operations to the feature map, The image processing device further comprises: an overall error calculation unit that calculates an overall error of the processing based on the output of the first conversion unit, the output of the second conversion unit, and the output of the third conversion unit, and a correct answer prepared in advance; an error propagation unit that calculates errors in the respective processes of the feature map generation unit, the first conversion unit, the second conversion unit, and the third conversion unit based on the error of the entire process; and A weight updating unit updates a weight coefficient used in a convolution operation of each of the processes based on an error in each of the processes.

2. The image processing device according to claim 1, characterized in that The object is an endoscope handling instrument.

3. The image processing device according to claim 1, characterized in that The object is a robotic arm.

4. The image processing device according to claim 1, characterized in that: The information related to the direction includes information related to the direction of the front end of the object and the reliability of the direction.

5. The image processing device according to claim 4, characterized in that: The image processing device further includes a comprehensive score calculation unit that calculates a comprehensive score of the candidate region based on the likelihood indicated by the second output and the reliability of the direction.

6. The image processing device according to claim 5, characterized in that: The information related to the reliability of the direction included in the information related to the direction is the magnitude of the direction vector indicating the direction of the front end of the object, The comprehensive score is a weighted sum of the likelihood and the direction vector.

7. The image processing device according to claim 5 or 6, characterized in that: The image processing device further includes a candidate region determination unit that determines a candidate region where the front end of the object is located based on the comprehensive score.

8. The image processing device according to claim 1, characterized in that: The information on the candidate region includes a positional displacement amount for bringing a reference point of the corresponding initial region closer to the front end of the object.

9. The image processing device according to claim 1, characterized in that: The image processing device also includes a candidate area deletion unit, which calculates the similarity between the first candidate area and the second candidate area among the candidate areas, and decides whether to delete either the first candidate area or the second candidate area based on the similarity and information related to the direction corresponding to the first candidate area and the second candidate area.

10. The image processing device according to claim 9, characterized in that: The similarity is the reciprocal of the distance between the first candidate region and the second candidate region.

11. The image processing device according to claim 9, characterized in that: The similarity is the degree of overlap between the first candidate region and the second candidate region.

12. An image processing device for detecting the front end of an object from an image, characterized in that: The image processing device comprises: An image input unit that receives image input; a feature map generating unit, which generates a feature map by applying a convolution operation to the image; a first transformation unit for generating a first output by applying a first transformation to the feature map; a second transformation unit that generates a second output by applying a second transformation to the feature map; and a third transformation unit that generates a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate points on the image, The second output indicates the likelihood of whether the front end of the object exists near the candidate point. The third output represents information related to the direction of the front end of the object that exists near the candidate point. The first conversion unit, the second conversion unit and the third conversion unit respectively apply convolution operations to the feature map, The image processing device further comprises: an overall error calculation unit that calculates an overall error of the processing based on the output of the first conversion unit, the output of the second conversion unit, and the output of the third conversion unit, and a correct answer prepared in advance; an error propagation unit that calculates errors in the respective processes of the feature map generation unit, the first conversion unit, the second conversion unit, and the third conversion unit based on the error of the entire process; and A weight updating unit updates a weight coefficient used in a convolution operation of each of the processes based on an error in each of the processes.

13. The image processing device according to claim 12, characterized in that: The object is an endoscope handling instrument.

14. The image processing device according to claim 12, characterized in that: The object is a robotic arm.

15. The image processing device according to claim 12, characterized in that: The information related to the direction includes information related to the direction of the front end of the object and the reliability of the direction.

16. The image processing device according to claim 15, characterized in that: The image processing device further includes a comprehensive score calculation unit that calculates a comprehensive score of the candidate point based on the likelihood indicated by the second output and the reliability of the direction.

17. The image processing device according to claim 16, characterized in that: The information related to the reliability of the direction included in the information related to the direction is the magnitude of the direction vector indicating the direction of the front end of the object, The comprehensive score is a weighted sum of the likelihood and the direction vector.

18. An image processing method for detecting the front end of an object from an image, characterized in that: The image processing method comprises: An image input step, accepting image input; a feature map generating step, generating a feature map by applying a convolution operation to the image; A first transformation step of generating a first output by applying a first transformation to the feature map; a second transformation step of generating a second output by applying a second transformation to the feature map; and a third transformation step of generating a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate regions on the image, The second output indicates the likelihood of whether the front end of the object exists in the candidate area. The third output represents information related to the direction of the front end of the object existing in the candidate area, The first conversion step, the second conversion step and the third conversion step respectively apply convolution operations to the feature map, The image processing method further comprises: an overall error calculation step, which calculates an overall error of the processing based on the output of the first conversion step, the output of the second conversion step, the output of the third conversion step, and a correct answer prepared in advance; an error propagation step of calculating errors in each of the processes of the feature map generation step, the first conversion step, the second conversion step, and the third conversion step based on the error of the entire process; and A weight updating step is performed to update the weight coefficients used in the convolution operation of each process based on the errors in each process.

19. A recording medium having a program recorded thereon, the program being used to detect a front end of an object from an image, wherein: The procedure allows the following steps to be performed: Accept image input; generating a feature map by applying a convolution operation to the image; generating a first output by applying a first transformation to the feature map; generating a second output by applying a second transformation to the feature map; and generating a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate regions on the image, The second output indicates the likelihood of whether the front end of the object exists in the candidate area. The third output represents information related to the direction of the front end of the object existing in the candidate area, The first transformation, the second transformation and the third transformation respectively apply convolution operations to the feature maps, The program also allows the following steps to be performed: Calculating an error of the entire process based on the output of the first conversion, the output of the second conversion, and the output of the third conversion, and a correct answer prepared in advance; calculating errors in each process of generating the feature map, the first conversion, the second conversion, and the third conversion based on the error of the entire process; and Based on the error in each of the processes, the weight coefficient used in the convolution operation of each of the processes is updated.

20. An image processing method for detecting the front end of an object from an image, characterized in that: The image processing method comprises: An image input step, accepting image input; a feature map generating step, generating a feature map by applying a convolution operation to the image; A first transformation step of generating a first output by applying a first transformation to the feature map; a second transformation step of generating a second output by applying a second transformation to the feature map; and a third transformation step of generating a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate points on the image, The second output indicates the likelihood of whether the front end of the object exists at the candidate point. The third output represents information related to the direction of the front end of the object existing at the candidate point, The first conversion step, the second conversion step and the third conversion step respectively apply convolution operations to the feature map, The image processing method further comprises: an overall error calculation step, which calculates an overall error of the processing based on the output of the first conversion step, the output of the second conversion step, the output of the third conversion step, and a correct answer prepared in advance; an error propagation step of calculating errors in each of the processes of the feature map generation step, the first conversion step, the second conversion step, and the third conversion step based on the error of the entire process; and A weight updating step is performed to update the weight coefficients used in the convolution operation of each process based on the errors in each process.

21. A recording medium having a program recorded thereon, the program being used to detect a front end of an object from an image, wherein: The procedure allows the following steps to be performed: Accept image input; generating a feature map by applying a convolution operation to the image; generating a first output by applying a first transformation to the feature map; generating a second output by applying a second transformation to the feature map; and generating a third output by applying a third transformation to the feature map, The first output represents information related to a predetermined number of candidate points on the image, The second output indicates the likelihood of whether the front end of the object exists at the candidate point. The third output represents information related to the direction of the front end of the object existing at the candidate point, The first transformation, the second transformation and the third transformation respectively apply convolution operations to the feature maps, The program also allows the following steps to be performed: Calculating an error of the entire process based on the output of the first conversion, the output of the second conversion, and the output of the third conversion, and a correct answer prepared in advance; calculating errors in each process of generating the feature map, the first conversion, the second conversion, and the third conversion based on the error of the entire process; and Based on the error in each of the processes, the weight coefficient used in the convolution operation of each of the processes is updated.

Images