Programs, methods, and apparatus

Abstract

To provide an image recognition system, an image recognition method, and a non-transitory computer-readable medium that improve recognition accuracy of an object included in an image.SOLUTION: An image recognition system (10) includes a first detection unit (101), an extracted image generation unit (104) and a second detection unit (107). The first detection unit (101) detects a person area that represents at least a portion of the body of a person from a first image obtained by imaging a target object related to the person. The extracted image generation unit (104) cuts out the extraction area set according to the person area from the first image. The second detection unit (107) detects the target object on the basis of the cut-out extraction area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image recognition system, an image recognition method, and a non-transitory computer-readable medium. [Background technology]

[0002] There are known techniques for detecting objects from captured images generated by a camera. For example, Patent Document 1 discloses an information processing device that uses a trained convolutional neural network (CNN) to detect belongings such as bags carried by people in a captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 207721 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, the captured image generated by the camera is reduced to a predetermined image size and input to the input layer of the trained CNN. If the detected belongings appear small in the captured image or if the belongings are long and thin, such as a white cane, the reduction may cause pixel crushing in the image area representing the belongings. Therefore, the information processing device described in Patent Document 1 above is unable to detect the belongings, and further improvement in object recognition accuracy is required.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide an image recognition system, an image recognition method, and a non-transitory computer-readable medium that can improve the accuracy of recognizing objects included in images. [Means for solving the problem]

[0006] An image recognition system according to one aspect of the present disclosure includes a first detection unit, an extracted image generation unit, and a second detection unit. The first detection unit detects a person area representing at least a part of the body of a person from a first image in which a target object related to the person is captured. The extracted image generation unit cuts out an extracted area set according to the person area from the first image. The second detection unit detects the target object based on the cut-out extracted area.

[0007] An image recognition method according to one aspect of the present disclosure includes a first detection step, an extraction image generation step, and a second detection step. The first detection step detects a person area representing at least a part of the body of a person from a first image in which a target object related to the person is captured. The extraction image generation step cuts out an extraction area set according to the person area from the first image. The second detection step detects the target object based on the cut-out extraction area.

[0008] A non-transitory computer-readable medium according to one aspect of the present disclosure stores an image recognition program for causing a computer to execute an image recognition method. The image recognition method includes a first detection step, an extracted image generation step, and a second detection step. The first detection step detects a person region representing at least a part of the person's body from a first image in which a target object related to the person is captured. The extracted image generation step cuts out an extracted region set according to the person region from the first image. The second detection step detects the target object based on the cut-out extracted region. [Effects of the Invention]

[0009] The present disclosure makes it possible to provide an image recognition system, an image recognition method, and a non-transitory computer-readable medium that can improve the accuracy of recognizing objects included in images. [Brief explanation of the drawings]

[0010] [Figure 1]1 is a block diagram showing a configuration of an image recognition system according to a first embodiment. [Figure 2] FIG. 10 is a block diagram showing the configuration of an image recognition system according to a second embodiment. [Figure 3] 10 is a flowchart showing the processing of the image recognition system according to the second embodiment. [Figure 4] FIG. 10 is a diagram for explaining the processing of the image recognition system according to the second embodiment. [Figure 5] FIG. 10 is a diagram for explaining the processing of the image recognition system according to the second embodiment. [Figure 6] FIG. 10 is a diagram for explaining the processing of the image recognition system according to the second embodiment. [Figure 7] FIG. 10 is a diagram for explaining the processing of the image recognition system according to the second embodiment. [Figure 8] FIG. 10 is a diagram for explaining the processing of the image recognition system according to the second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a display of the image recognition system according to the second embodiment. [Figure 10] FIG. 10 is a block diagram showing the configuration of an image recognition system according to a third embodiment. [Figure 11] 11 is a flowchart showing the processing of the image recognition system according to the third embodiment. [Figure 12] FIG. 10 is a block diagram showing the configuration of an image recognition system according to a fourth embodiment. [Figure 13] 10 is a flowchart showing the processing of the image recognition system according to the fourth embodiment. [Figure 14] FIG. 10 is a block diagram showing the configuration of an image recognition system according to a fifth embodiment. [Figure 15] 13 is a flowchart showing an object region detection process by a second detection unit according to the fifth embodiment. [Figure 16] 13A and 13B are diagrams for explaining the object region detection process of the second detection unit according to the fifth embodiment. [Figure 17] FIG. 13 is a block diagram showing the configuration of an image recognition system according to a sixth embodiment. [Figure 18] 13 is a flowchart showing a person area detection process of a first detection unit according to the sixth embodiment. [Figure 19] 13 is a diagram for explaining a person area detection process by a first detection unit according to the sixth embodiment. FIG. [Figure 20] FIG. 13 is a block diagram showing the configuration of an image recognition system according to a seventh embodiment. [Figure 21] FIG. 20 is a diagram showing an example of a data structure of region selection information according to the seventh embodiment. [Figure 22] 13 is a flowchart showing a person area detection process of a first detection unit according to the seventh embodiment. [Figure 23] FIG. 1 is a configuration diagram of a computer according to first to seventh embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present disclosure will be described below through embodiments, but the disclosure according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily required as means for solving the problems. In each drawing, the same elements are given the same reference numerals, and redundant description is omitted as necessary. In the following description, when the number of pixels in an image (also referred to as image size) or the number of pixels in an image area is expressed as X×Y, X indicates the number of pixels in the width direction of the rectangular image or rectangular image area, and Y indicates the number of pixels in the height direction of the rectangular image or rectangular image area. Furthermore, X and Y are natural numbers.

[0012] <Embodiment 1> First, a first embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an image recognition system 10 according to the first embodiment. The image recognition system 10 includes a first detection unit 101, an extracted image generation unit 104, and a second detection unit 107. The first detection unit 101 detects a person area representing at least a part of the body of a person from a first image in which a target object related to a person is captured. The extracted image generating unit 104 cuts out an extracted area set in accordance with the person area from the first image. The second detection unit 107 detects a target object based on the cut-out extracted region.

[0013] As described above, according to the configuration of the first embodiment, the image recognition system 10 detects a target object based on an extracted region excluding unnecessary regions. Therefore, when converting a target image to have an image size equivalent to that of an input image for CNN in order to detect the target object, it is possible to avoid pixel collapse in the image region representing the target object. This improves the recognition accuracy of objects included in the image.

[0014] <Embodiment 2> Next, a second embodiment of the present disclosure will be described with reference to Figures 2 to 9. Figure 2 is a block diagram showing the configuration of an image recognition system 20 according to the second embodiment. The image recognition system 20 is a computer system that recognizes a target object related to a person from a captured image. The image recognition system 20 includes an acquisition unit 200, a first detection unit 201, an extracted image generation unit 204, a second detection unit 207, and a storage unit 211.

[0015] The acquisition unit 200 acquires various information necessary for a series of recognition processes such as detection of a target object. The acquisition unit 200 acquires a first image I1 from an imaging device (not shown) such as a surveillance camera or by receiving input from a user. The first image I1 captures an image of a person and a target object related to the person. The target object may be the person's belongings. The person's belongings are not limited to objects held by the person, but may include objects carried in some form. The person's belongings may be an object held by the person in their hand (such as a white cane or bag), an object hung from their neck (such as an identification card), an object worn on their head (such as a hat), or an object worn on their face (such as glasses). The acquisition unit 200 supplies the acquired information to the first detection unit 201.

[0016] The first detection unit 201 detects a person region representing at least a part of a person's body from the first image I1 using a person region detector. The person region may represent the entire person or a part of the person's body. For example, the part of the person's body may be a part such as a hand, arm, neck, head, or face. The person region detector may be, for example, a detector that has been trained in advance to detect an image region representing a person or an image region representing a specific part of a person from an input image. Such a detector may use any existing object detection model. In the second embodiment, the person region detector uses a detection model such as an SSD (Single Shot MultiBox Detector) or a WSSD (Weighted Single Shot MultiBox Detector) that includes a trained CNN. However, the present invention is not limited to this, and the person region detector may also use any other detection model such as an SVM (Support Vector Machine). The first detection unit 201 supplies information about the detected person area to the extracted image generation unit 204.

[0017] The extraction image generation unit 204 cuts out an extraction area from the first image I1. Here, the extraction area is an image area set according to the person area detected by the first detection unit 201. For example, the extraction area may be an image area arranged in a predetermined position with respect to the person area in the first image I1 and within a predetermined range. Alternatively, the extraction area may be an image area arranged in a predetermined range with the person area at its center in the first image I1. As an example, the extraction area may be an image area centered on the center of the person area in the first image I1 and having a number of pixels set based on the number of pixels of the person area. Alternatively, the extraction area may be the same image area as the person area.

[0018] The extracted image generation unit 204 then generates a second image I2 based on the extracted region. The extracted image generation unit 204 performs image conversion on the extracted region so that the size of the extracted region is equivalent to the image size of an input image of an object detector used in the second detection unit 207 (described later), thereby generating the second image I2. The extracted image generating unit 204 supplies the generated second image I2 to the second detecting unit 207.

[0019] The second detection unit 207 detects an object region, which is an image region representing a target object, from the second image I2 using an object detector. Here, the object detector may be a detector that has been trained in advance to detect an image region representing a target object from an input image. The object detector may use a detection model such as SSD or WSSD including a trained CNN.

[0020] The storage unit 211 is a storage medium that stores various information necessary for a series of recognition processes such as target object detection, etc. For example, the storage unit 211 stores trained parameters of the person area detector used by the first detection unit 201 and trained parameters of the object detector used by the second detection unit 207.

[0021] Next, an image recognition method of the image recognition system 20 will be described using Fig. 3 with reference to Figs. 4 to 8. Fig. 3 is a flowchart showing the processing of the image recognition system 20 according to the second embodiment. Figs. 4 to 8 are diagrams for explaining the processing of the image recognition system 20 according to the second embodiment.

[0022] First, in step S10, the acquisition unit 200 acquires a first image I1. Here, the first image I1 is an image of a person and a target object related to the person, as shown in FIG. 4. In this example, the target object is a "white cane" held by the person. As shown in FIG. 4, the first image I1 has a pixel count of X1 x Y1. In this example, the first image I1 is a full high-definition image, i.e., X1 is 1920 and Y1 is 1080.

[0023] Next, in step S11, the first detection unit 201 detects a person area from the first image I1. In this example, the first detection unit 201 acquires the learned parameters of the person area detector from the storage unit 211, and uses the learned person area detector including the learned parameters to detect a person area P that is the entire person from the first image I1. As shown in FIG. 5, the person area P has the number of pixels of XP1×YP1 (XP1<X1 and YP1<Y1).

[0024] At this time, the first detection unit 201 may convert the first image I1 to the same image size as the image size of the input image of the person area detector (for example, 300 pixels×300 pixels), and input the converted image to the person area detector. Then, the first detection unit 201 may specify the person area P in the first image I1 based on the image area related to the output result. Note that when the first detection unit 201 performs image conversion, it may perform the same processing as the image conversion processing performed by the extraction image generation unit 204 in step S13 described later. Then, the first detection unit 201 supplies information related to the person area P to the extraction image generation unit 204.

[0025] Next, in step S12, the extraction image generation unit 204 specifies an extraction area based on the person area. For example, as shown in FIG. 6, the extraction image generation unit 204 specifies an image area having the number of pixels of XA1×YA1 as the extraction area A with the center of the person area P as the center (XP1<XA1<X1 and YP1<YA1<Y1). Here, XA1 and YA1 may be set based on XP1 and YP1, respectively. For example, XA1 and YA1 may each have a value that is N times (N>1) of XP1 and YP1. N may be determined in advance.

[0026] Next, in step S13, the extracted image generation unit 204 generates a second image I2 based on the identified extracted region. For example, as shown in FIG. 7, the extracted image generation unit 204 cuts out an extracted region A having XA1×YA1 pixels from the first image I1 and converts the extracted region A into a second image I2 having X2×Y2 pixels. X2 and Y2 are equal to the number of pixels in the width direction and the number of pixels in the height direction, respectively, of the image size of the input image of the object detector. X2 and Y2 may have values ​​smaller than X1 and Y1, respectively. In this example, X2 and Y2 are both 300.

[0027] At this time, the extraction image generation unit 204 performs image conversion processing such as enlarging, reducing, stretching, or compressing the extraction area A based on the number of pixels of the extraction area A and the second image I2. In this example, the extraction image generation unit 204 performs image conversion processing on the extraction area A by a factor of (X2 / XA1) in the width direction and (Y2 / YA1) in the height direction. For example, when enlarging or reducing the extraction area A, the extraction image generation unit 204 may change the spacing between a specific pixel included in the extraction area A and the surrounding pixels and interpolate the pixels therebetween. When expanding the extraction area A, the extraction image generation unit 204 may increase the spacing between pixels in the direction of expansion and interpolate the pixels therebetween. When compressing the extraction area A, the extraction image generation unit 204 may decrease the spacing between pixels in the direction of contraction and interpolate the pixels as appropriate. When expanding or extending the extraction region A, the extraction image generation unit 204 may perform zero padding in the direction of expansion or extension instead of widening the spacing between pixels. The extracted image generating unit 204 then supplies the generated second image I2 to the second detecting unit 207.

[0028] Next, in step S14, the second detection unit 207 obtains the trained parameters of the object detector from the storage unit 211 and detects an object region from the second image I2 using the trained object detector including the trained parameters. As described above, the image size of the input image to the object detector is equal to X2 × Y2. Therefore, in this example, the image size is 300 × 300. For example, as shown in Figure 8, the second detection unit 207 detects an object region B representing a white cane in the second image I2.

[0029] Next, in step S15, second detection unit 207 determines whether or not to end the series of processes. If so, second detection unit 207 ends the processes, and if not, returns the process to step S11.

[0030] If steps S11 to S13 are not performed as in the information processing device described in Patent Document 1, the object detector receives the first image I1, which has been reduced from X1 × Y1 (= 1920 × 1080) to X2 × Y2 (= 300 × 300), as the second image I2. In other words, the first image I1 is reduced so that the number of pixels in the width direction is (X2 / X1) (= approximately 0.16) times larger and the number of pixels in the height direction is (Y2 / Y1) (= approximately 0.28) times larger. Here, if the ratio of the resolution of the converted image to the image before conversion is S, then the width direction S is (X2 / X1) (= approximately 0.16) and the height direction S is (Y2 / Y1) (= approximately 0.28), respectively. This significantly reduces the image resolution due to the image conversion process. This makes pixel crushing more likely to occur in the object region of the second image I2. However, according to the second embodiment, since the extracted region A clipped in steps S11 to S13 is converted into the second image I2, the resolution loss due to image conversion is reduced as the number of pixels in the extracted region A approaches that of the second image I2. For example, if the number of pixels in the extracted region A is approximately equal to X2 × Y2 (= 300 × 300), the resolution of the generated second image I2 is maintained at approximately the same level as that of the extracted region A in the first image I1. This prevents pixel crushing in the object region, thereby improving the recognition accuracy of the target object included in the first image I1. Furthermore, since the extracted region A has information removed from unnecessary areas in the first image I1, it is possible to improve recognition accuracy while reducing computational costs. The image recognition system 20 according to the second embodiment is particularly effective when the target object is a "white cane," which has significantly fewer pixels in either width or height than the other. Therefore, the image recognition system 20 can be applied to systems such as a voice assistance system that uses surveillance camera footage to identify a person with low vision carrying a "white cane" and provides voice guidance to the identified person.

[0031] In the second embodiment, the image size of the input image to the object detector is 300 × 300, i.e., X2 and Y2 are both 300. However, X2 and Y2 may both be less than 300. That is, the image size of the input image to the object detector may be 200 × 200, 150 × 150, or 100 × 100. In this case, the object detector may be a detector trained on an input image of a predetermined image size. However, when the input image size of the object detector is such a small value, the resolution of the generated second image I2 is lower than when the image size is 300 × 300. However, even in such cases, the impact of reduced resolution on recognition accuracy can be mitigated by setting X2 and Y2 so that the ratio S of the resolutions of the images before and after conversion is larger by a predetermined amount than S when the object region is detected without performing steps S11 to S13. Moreover, in this case, the second detection unit 207 can detect the object region using a lighter detector, thereby significantly reducing computational costs while maintaining a certain level of recognition accuracy.

[0032] The image recognition system 20 of the second embodiment may further include a display unit. Fig. 9 is a diagram showing an example of the display of the image recognition system 20 according to the second embodiment. As shown in this figure, the display unit may display a first image I1 and a second image I2.

[0033] In response to detection of object region B from second image I2, the display unit may superimpose and display object region B on second image I2. The display unit may also display an extracted region A corresponding to the detected object region B superimposed on the first image I1.

[0034] Furthermore, as shown in the figure, the display unit may have an image input means for receiving input of a first image I1 from a user. The image input means may be connected to the acquisition unit 200. Then, the display unit may display the first image I1 in response to receiving input of the first image I1 from the image input means.

[0035] The display unit may have an image output means for receiving a request from the user to output the first image I1 with the extracted area A superimposed thereon, the second image I2, or the second image I2 with the object area B superimposed thereon. In response to the image output means receiving the output request, the image recognition system 20 may output the requested image data in a predetermined data format.

[0036] <Embodiment 3> Next, a third embodiment of the present disclosure will be described with reference to Figures 10 to 11. The third embodiment is characterized in that the image size of the second image I2 is determined based on the size of the extraction region. 10 is a block diagram showing the configuration of an image recognition system 30 according to embodiment 3. The image recognition system 30 according to embodiment 3 basically has the same configuration and functions as the image recognition system 20 according to embodiment 2. However, the image recognition system 30 according to embodiment 3 differs from embodiment 2 in that it includes an extracted image generation unit 304 instead of the extracted image generation unit 204.

[0037] The extracted image generating unit 304 has the same configuration and functions as the extracted image generating unit 204 , and further includes a size determining unit 305 . The size determination unit 305 determines the number of pixels that the second image I2 will have based on the number of pixels that the extraction region has. For example, the size determination unit 305 selects one of the predetermined numbers of pixels that the second image I2 may have based on the number of pixels that the extraction region has, and determines the selected number of pixels as the number of pixels that the second image I2 will have. The number of pixels that the second image I2 may have may include, for example, 300 x 300 and 200 x 200. The extraction image generation unit 204 then converts the extraction region to match the determined number of pixels and generates the second image I2.

[0038] Note that the storage unit 211 of the third embodiment stores learned parameters of an object detector for each number of pixels that the second image I2 may have, in addition to the information stored in the storage unit 211 of the second embodiment. For example, the storage unit 211 stores learned parameters of an object detector for an input image with an image size of 300 × 300, and learned parameters of an object detector for an input image with an image size of 200 × 200.

[0039] Fig. 11 is a flowchart showing the processing of the image recognition system 30 according to the third embodiment. The steps shown in Fig. 11 include steps S20 to S24 instead of steps S13 to S14 shown in Fig. 3. Note that steps similar to those shown in Fig. 3 are given the same reference numerals and their explanations will be omitted.

[0040] In step S20, the size determination unit 305 of the extraction image generation unit 304 determines the number of pixels of the second image I2 based on the number of pixels of the extraction area A, in response to the extraction image generation unit 304 identifying the extraction area A in step S12.

[0041] For example, if the number of pixels in at least one of the height and width directions of the extracted area A is less than 300, the size determination unit 305 may determine the number of pixels in the second image I2 to be 200 × 200. If the number of pixels in both the height and width directions is 300 or more, the size determination unit 305 may determine the number of pixels in the second image I2 to be 300 × 300.

[0042] The size determination unit 305 may also determine the number of pixels in the second image I2 so that the ratio S of the resolution of the images before and after conversion is greater than a predetermined reference value S0. Here, the ratio S of the resolution of the images before and after conversion is a value obtained by dividing the number of pixels in the second image I2 by the number of pixels in the extraction area A (i.e., X2 / XA1 in the width direction or Y2 / YA1 in the height direction). For example, to ensure the resolution of the second image I2, the size determination unit 305 determines, from the number of pixels that the second image I2 can have, a number of pixels that is greater than the product of the number of pixels in the extraction area A and the reference value S0, as the number of pixels in the second image I2. Specifically, the size determination unit 305 selects, from the values ​​of X2 (or Y2) that the second image I2 can have, the one that is greater than the product of the smaller of the number of pixels in the height and width directions of the extraction area A and the reference value S0, and determines this as X2 (or Y2) of the second image I2. If there are multiple values ​​of X2 (or Y2) selected in this way, the size determination unit 305 may determine the smallest value among these values ​​as X2 (or Y2) of the second image I2, thereby reducing calculation costs.

[0043] In step S22, the extraction image generation unit 304 performs the same process as in step S13 based on the identified extraction area A and the determined number of pixels of the second image I2 to generate the second image I2. The extraction image generation unit 304 then supplies the generated second image I2 to the second detection unit 207.

[0044] In step S24, the second detection unit 207 acquires trained parameters of the object detector corresponding to the determined number of pixels of the second image I2 from the storage unit 211. Then, the second detection unit 207 detects the object region B from the second image I2 using the trained object detector including the trained parameters.

[0045] As described above, according to the third embodiment, the image recognition system 30 determines the image size of the second image I2 based on the size of the extracted region A to be cut out. Therefore, it is possible to ensure the resolution of the input image in detecting the object region, thereby ensuring recognition accuracy and reducing calculation costs.

[0046] <Embodiment 4> Next, a fourth embodiment of the present disclosure will be described with reference to Figures 12 and 13. Here, if a person appears small in the first image I1, it is highly likely that a target object related to that person also appears small. In such a case, it becomes difficult to accurately detect the object region of the target object. The fourth embodiment is characterized in that if the person region P is smaller than a predetermined size, the subsequent object region detection process is not performed.

[0047] 12 is a block diagram showing the configuration of an image recognition system 40 according to embodiment 4. The image recognition system 40 according to embodiment 4 basically has the same configuration and functions as the image recognition system 30 according to embodiment 3. However, the image recognition system 40 according to embodiment 4 differs from embodiment 3 in that it includes an extracted image generation unit 404 instead of the extracted image generation unit 304.

[0048] The extracted image generating unit 404 has the same configuration and functions as the extracted image generating unit 304 , and also has a determining unit 406 . The determination unit 406 determines whether to generate a second image I2 based on the number of pixels in the person area P. That is, the determination unit 406 determines whether to proceed to the subsequent object area detection step based on the number of pixels in the person area P.

[0049] Fig. 13 is a flowchart showing the processing of the image recognition system 40 according to the fourth embodiment. The steps shown in Fig. 13 include step S30 in addition to the steps shown in Fig. 11. Note that steps similar to those shown in Fig. 11 are given the same reference numerals and their explanations will be omitted.

[0050] In step S30, in response to the first detection unit 201 detecting the person region P from the first image I1 in step S11, the determination unit 406 of the extracted image generation unit 404 determines whether the number of pixels in the person region P is greater than a predetermined first threshold. Specifically, the determination unit 406 determines whether the number of pixels in the height or width direction of the person region P is greater than the first threshold. If the determination unit 406 determines that the number of pixels is greater than the first threshold (Yes in step S30), the process proceeds to step S12. On the other hand, if the determination unit 406 determines that the number of pixels is not greater than the first threshold (No in step S30), the process proceeds to step S15.

[0051] As described above, according to the fourth embodiment, the image recognition system 40 determines whether to generate the second image I2 based on the number of pixels in the person area P, and therefore, if the person area P is smaller than a predetermined size, subsequent processing can be omitted. This reduces calculation costs and ensures that the series of recognition processes can be performed in real time.

[0052] Note that the determination unit 406 may determine whether or not to generate the second image I2 based on the number of pixels in the extraction area A, instead of the number of pixels in the person area P. In this case, step S30 shown in FIG. 13 is omitted. Then, in step S12, the extraction image generation unit 404 identifies the extraction area A based on the person area P and determines whether the number of pixels in the extraction area A is greater than a first threshold. If the determination unit 406 determines that the number of pixels is greater than the first threshold, the process proceeds to step S20. On the other hand, if the determination unit 406 determines that the number of pixels is not greater than the first threshold, the process proceeds to step S15. In this case, the same effect as described above can be obtained.

[0053] <Embodiment 5> Next, a fifth embodiment of the present disclosure will be described with reference to Figures 14 to 16. The fifth embodiment is characterized in that an object region is identified from the detection results of an object detector based on relative position information between the object region and a person region.

[0054] 14 is a block diagram showing the configuration of an image recognition system 50 according to the fifth embodiment. The image recognition system 50 according to the fifth embodiment has basically the same configuration and functions as the image recognition system 40 according to the fourth embodiment. However, the image recognition system 50 according to the fifth embodiment differs from the fourth embodiment in that the image recognition system 50 includes a second detection unit 507 instead of the second detection unit 207.

[0055] The second detection unit 507 basically has the same functions as the second detection unit 207 , but also includes a candidate area detection unit 508 and a specification unit 509 .

[0056] The candidate region detection unit 508 uses an object detector to detect one or more candidate regions in the second image I2, where a candidate region is an image region that is presumed to represent the target object.

[0057] The identification unit 509 identifies an object region B from one or more candidate regions based on relative position information of the one or more candidate regions with respect to the person region P. In the fifth embodiment, the relative position information may be, for example, the distance between the person region P and the candidate region.

[0058] Next, the object region detection processing of the second detection unit 507 according to the fifth embodiment will be described using Fig. 15 with reference to Fig. 16. Fig. 15 is a flowchart showing the object region detection processing of the second detection unit 507 according to the fifth embodiment. Fig. 16 is a diagram for explaining the object region detection processing of the second detection unit 507 according to the fifth embodiment.

[0059] First, in step S40, the candidate area detection unit 508 of the second detection unit 507 obtains trained parameters of an object detector that correspond to the number of pixels in the second image I2 from the storage unit 211. The candidate area detection unit 508 then detects candidate areas from the second image I2 using the trained object detector that includes the trained parameters. As shown in Fig. 16, the candidate area detection unit 508 detects multiple candidate areas C1 and C2 included in the second image I2.

[0060] Next, in step S42, the identification unit 509 calculates the distance between each candidate area and a person area P2 in the second image I2. Here, the person area P2 in the second image I2 is an image area corresponding to the person area P in the first image I1. As shown in Fig. 16, the distances d1 and d2 between each candidate area C1, C2 and the person area P2 may be the distance between a representative point such as the center of the candidate area and a representative point such as the center of the person area P2.

[0061] The identification unit 509 then determines whether or not there is a candidate area whose distance to the person area P2 is less than a second threshold. The second threshold may be a predetermined value or a value set according to the number of pixels in the person area P2. If the identification unit 509 determines that there is a candidate area whose distance to the person area P2 is less than the second threshold (Yes in step S42), the identification unit 509 proceeds to step S44. On the other hand, if there is not (No in step S42), the identification unit 509 returns the process to step S15 shown in FIG. 13.

[0062] Then, in step S44, the identification unit 509 identifies the candidate area whose distance to the person area P2 is less than the second threshold as the object area B. Then, the identification unit 509 returns the process to step S15 shown in FIG.

[0063] According to the fifth embodiment, the image recognition system 50 determines whether an object is a person's possession based on the distance to the person, and therefore can properly detect a target object held by the person, thereby improving recognition accuracy. For example, if the target object is a "white cane," the image recognition system 50 can properly detect the white cane using video from a surveillance camera, and further, can properly identify the person holding the white cane.

[0064] If the second threshold is dynamically set according to the number of pixels in the person area P2, the image recognition system 50 will be able to more appropriately detect the target object held by the person, thereby further improving the recognition accuracy.

[0065] In the fifth embodiment, the image recognition system 50 identifies a candidate area whose distance to the person area P2 is less than the second threshold as the object area B. However, instead, the image recognition system 50 may identify a candidate area whose distance to the person area P2 is the smallest as the object area B. This can also achieve the same effect as described above.

[0066] <Embodiment 6> Next, a sixth embodiment of the present disclosure will be described with reference to Figures 17 to 19. The sixth embodiment is characterized in that a person region is detected based on the skeletal structure of the person. 17 is a block diagram showing the configuration of an image recognition system 60 according to the sixth embodiment. The image recognition system 60 according to the sixth embodiment basically has the same configuration and functions as the image recognition system 50 according to the fifth embodiment. However, the image recognition system 60 according to the sixth embodiment differs from the fifth embodiment in that it includes a first detection unit 601 instead of the first detection unit 201.

[0067] The first detection unit 601 has basically the same function as the first detection unit 201, and detects a person area P from the first image I1. In this sixth embodiment, the person area P represents a part of the person's body. For example, the person area P represents a part such as the hand, neck, head, or face. The first detection unit 601 has a skeleton estimation unit 602.

[0068] The skeleton estimation unit 602 estimates a two-dimensional skeleton structure of a person using a skeleton estimation model. Then, the skeleton estimation unit 602 detects a person area P based on the estimated two-dimensional skeleton structure. The skeleton estimation model may be an existing skeleton estimation model that has been trained by machine learning.

[0069] Next, the person area detection processing of the first detection unit 601 according to the sixth embodiment will be described using Fig. 18 with reference to Fig. 19. Fig. 18 is a flowchart showing the person area detection processing of the first detection unit 601 according to the sixth embodiment. Fig. 19 is a diagram for explaining the person area detection processing of the first detection unit 601 according to the sixth embodiment.

[0070] First, in step S50, the skeleton estimation unit 602 of the first detection unit 601 estimates a two-dimensional skeletal structure of a person from the first image I1 using a skeleton estimation model. The estimated two-dimensional skeletal structure here is composed of key points, which are characteristic points such as joints, and bones that connect the key points. For example, the skeleton estimation unit 602 first extracts feature points that can be key points from the first image I1 and detects each key point of the person by referring to information obtained by machine learning of the images of the key points. In the example shown in FIG. 19, the following key points of the person are detected: head K1, neck K2, right shoulder K31, left shoulder K32, right elbow K41, left elbow K42, right hand K51, left hand K52, right hip K61, left hip K62, right knee K71, left knee K72, right foot K81, and left foot K82.

[0071] Next, in step S52, the skeleton estimation unit 602 identifies a person region P based on the estimated two-dimensional skeleton structure. In the example shown in Fig. 19, the target object is a "white cane," and the person region P represents the person's "hand." Therefore, the skeleton estimation unit 602 may select a right hand K51, which is a key point related to the person's "hand," from the detected multiple key points, and identify a predetermined area including the right hand K51 as the person region P.

[0072] At this time, the skeleton estimation unit 602 may determine the extent of the person region P based on the lengths of the bones connecting the key points. For example, the skeleton estimation unit 602 may determine the extent of the person region P based on the length of the bone B41 connecting the right hand K51 and the right elbow K41, and identify the person region P. Then, the skeleton estimation unit 602 returns the process to step S30 shown in FIG.

[0073] In step S42 shown in FIG. 15, the identification unit 509 may use a point corresponding to the key point selected when estimating the person area P as a representative point of the person area P2 to calculate the distance between the candidate area and the person area P2.

[0074] Thus, according to embodiment 6, the image recognition system 60 detects a person area based on the person's skeletal structure, making it possible to more appropriately detect target objects held by the person based on the person area, thereby further improving recognition accuracy.

[0075] <Embodiment 7> Next, a seventh embodiment of the present disclosure will be described with reference to Figures 20 to 22. The seventh embodiment is characterized in that a person area is identified according to the type of target object. 20 is a block diagram showing the configuration of an image recognition system 60 according to the seventh embodiment. The image recognition system 70 according to the seventh embodiment basically has the same configuration and functions as the image recognition system 60 according to the sixth embodiment. However, the image recognition system 70 according to the seventh embodiment differs from the sixth embodiment in that it includes a first detection unit 701 instead of the first detection unit 601 and a storage unit 711 instead of the storage unit 211.

[0076] The first detection unit 701 has the same configuration and functions as the first detection unit 601 , and further has a region selection unit 703 . The body part selection unit 703 selects the type of body part based on the type of the target object. The skeleton estimation unit 602 identifies the person area P according to the selected type of body part.

[0077] The storage unit 711 stores part selection information that associates the type of target object with the type of part, in addition to the configuration and functions of the storage unit 211. The storage unit 711 also stores trained parameters of the human area detector for each type of part, and trained parameters of the object detector for each type of target object.

[0078] 21 is a diagram illustrating an example of a data structure of region selection information according to embodiment 7. The region selection information includes the type of target object and the type of region. The type of the target object may be a "white cane," a "bag," or a "hat," for example. The type of body part may be "hand" if the type of target object is a "white cane," may be "hand" or "arm" if the type of target object is a "bag," or may be "head" if the type of target object is a "hat."

[0079] Note that the part selection information may further include a value related to a second threshold used when identifying an object region, as shown in this figure. The value related to the second threshold included in the part selection information may be the value of the second threshold, or may be a normalized value of the second threshold when the person region P2 is normalized to a predetermined size. In this case, in step S42 shown in FIG. 15, the identification unit 509 may obtain a value related to the second threshold from the part selection information in the storage unit 711 based on the type of target object and the type of part, and set the value of the second threshold based on this value. The identification unit 509 may then determine whether there is a candidate region whose distance from the person region P2 is less than the second threshold.

[0080] FIG. 22 is a flowchart showing the person area detection process of the first detection unit 701 according to the seventh embodiment. First, in step S60, the part selection unit 703 of the first detection unit 701 acquires target object type information relating to the type of the target object via the acquisition unit 200. Note that the acquisition unit 200 may acquire the target object type information by accepting input from the user.

[0081] Next, in step S62, the part selection unit 703 refers to the part selection information in the storage unit 711 and selects the type of part associated with the type of the target object.

[0082] Next, in step S63, the skeleton estimation unit 602 of the first detection unit 701 estimates a two-dimensional skeleton structure of a person from the first image I1 using the skeleton estimation model, similar to step S50 shown in FIG.

[0083] Then, in step S64, the skeleton estimation unit 602 identifies a person area P based on the estimated two-dimensional skeleton structure and the type of the selected part.

[0084] As described above, according to the seventh embodiment, the image recognition system 70 identifies the person area P based on the type of the target object. This allows the image recognition system 70 to more appropriately detect the target object based on the person area P, thereby further improving the recognition accuracy.

[0085] In the above-described first to seventh embodiments, the computer is configured as a computer system including a personal computer, a word processor, etc. However, the present invention is not limited to this, and the computer can also be configured as a LAN (local area network) server, a computer (personal computer) communication host, a computer system connected to the Internet, etc. It is also possible to distribute functions to each device on the network and configure a computer as a whole network.

[0086] Although the disclosure has been described as a hardware configuration in the above-described first to seventh embodiments, the disclosure is not limited to this. The disclosure can also be realized by causing a processor 1010 (described later) to execute a computer program to perform the image recognition processes such as the person area detection process, the second image generation process, and the object area detection process.

[0087] Fig. 23 is an example of a configuration diagram of a computer 1900 according to embodiments 1 to 7. As shown in Fig. 23, the computer 1900 includes a control unit 1000 for controlling the entire system. An input device 1050, a storage device 1200, a storage medium drive device 1300, a communication control device 1400, and an input / output I / F 1500 are connected to the control unit 1000 via a bus line such as a data bus.

[0088] The control unit 1000 includes a processor 1010 , a ROM 1020 , and a RAM 1030 . The processor 1010 performs various information processing and control operations in accordance with programs stored in various storage units such as the ROM 1020 and the storage device 1200 . The ROM 1020 is a read-only memory that stores in advance various programs and data for the processor 1010 to perform various controls and calculations.

[0089] The RAM 1030 is a random access memory used as a working memory by the processor 1010. In this RAM 1030, various areas can be reserved for performing various processes according to the first to seventh embodiments.

[0090] The input device 1050 is an input device that accepts input from a user, such as a keyboard, a mouse, or a touch panel. For example, the keyboard is provided with various keys, such as a numeric keypad, function keys for executing various functions, and cursor keys. The mouse is a pointing device that is used to specify a corresponding function by clicking on a key or icon displayed on the display device 1100. The touch panel is an input device that is placed on the surface of the display device 1100, and identifies a user's touch position corresponding to various operation keys displayed on the screen of the display device 1100, and accepts input from the operation keys displayed corresponding to the touch position.

[0091] The display device 1100 may be, for example, a CRT or a liquid crystal display. This display device displays the results of inputs made using a keyboard or mouse, as well as the final image information that has been searched for. The display device 1100 also displays images of operation keys for performing various operations required from a touch panel in accordance with the various functions of the computer 1900.

[0092] The storage device 1200 is composed of a readable / writable storage medium and a drive for reading and writing various information such as programs and data from and to the storage medium. The storage medium used in this storage device 1200 is mainly a hard disk or the like, but may be any non-transitory computer readable medium that can be driven by a storage medium drive device 1300 described below. The storage device 1200 has a data storage unit 1210, a program storage unit 1220, and other storage units (not shown) (for example, storage units for backing up programs, data, etc. stored in the storage device 1200). The program storage unit 1220 stores programs for realizing various processes in the first to seventh embodiments. The data storage unit 1210 stores various data of various databases according to the first to seventh embodiments.

[0093] The storage medium drive device 1300 is a drive device that allows the processor 1010 to read data including computer programs and documents from an external storage medium (external storage medium). Here, the external storage medium refers to a non-transitory computer-readable medium that can non-temporarily store computer programs, data, and the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Various programs may also be supplied to a computer via various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply various programs to a computer via wired communication paths such as electric wires and optical fibers, wireless communication paths, and the storage medium drive 1300.

[0094] That is, in the computer 1900 , the processor 1010 of the control unit 1000 reads various programs from an external storage medium set in the storage medium drive device 1300 and stores them in each unit of the storage device 1200 .

[0095] When the computer 1900 executes various processes, it loads the corresponding programs from the storage device 1200 into the RAM 1030 and executes them. However, the computer 1900 can also load and execute programs directly from an external storage medium into the RAM 1030 using the storage medium drive device 1300, rather than from the storage device 1200. Depending on the computer, various programs may be stored in the ROM 1020 in advance and executed by the processor 1010. Furthermore, the computer 1900 may download various programs and data from other storage media via the communication control device 1400 and execute them.

[0096] The communication control device 1400 is a control device for networking the computer 1900 with various external electronic devices such as other personal computers and word processors, etc. The communication control device 1400 enables access to the computer 1900 from these various external electronic devices.

[0097] The input / output I / F1500 is an interface for connecting various input / output devices via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0098] Note that a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like may be used as the processor 1010. Furthermore, a plurality of these may be used in parallel.

[0099] The order of execution of each process in the systems and methods shown in the claims, specification, and drawings is not specifically stated as "before," "prior to," etc., and may be implemented in any order as long as the output of a previous process is not used in a subsequent process. Even if the operational flow in the claims, specification, and drawings is described using "first," "next," etc. for convenience, this does not mean that it is essential to implement the process in this order.

[0100] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention. [Explanation of symbols]

[0101] 10,20,30,40,50,60,70 Image Recognition System 101, 201, 601, 701 First detection unit 104,204,304,404 Extracted image generation unit 107,207,507 Second detection unit 200 Acquisition Department 211,711 storage section 305 Size determination section 406 Judgment section 508 Candidate region detection unit 509 Specific section 602 Skeleton Estimation Department 703 Part Selection Section I1 First image I2 Second image P People area A extraction area 1000 control section 1010 processor 1020 ROM 1030 RAM 1050 Input Device 1100 Display device 1200 storage device 1210 Data storage unit 1220 Program Storage Unit 1300 Storage media drive unit 1400 Communication Control Device 1500 Input / Output Interface 1900 Computer

Claims

1. Computer, a means for acquiring an image; a means for determining whether or not to execute processing for detecting an object appearing in the image based on the size of an area in the image that shows a person; means for executing the process when the result of the determination indicates that the process should be executed; A program that functions as a means for identifying the area of ​​an object appearing in the image based on relative position information between the area showing the person and the results of detection by the processing.

2. the means for determining whether or not to execute a process for detecting an object appearing in the image determines not to execute the process for detecting an object appearing in the image when the number of pixels in the area showing the person is equal to or less than a threshold value; The program according to claim 1.

3. The process for detecting an object in the image includes a process for detecting an object other than the person from a part of the image. The program according to claim 1 or 2.

4. Acquire an image, determining whether to execute a process for detecting an object appearing in the image based on the size of an area in the image that shows a person; If the result of the determination indicates that the process is to be executed, execute the process; identifying an area of ​​the object in the image based on relative position information between the area and the person from the detection results of the processing; method.

5. a means for acquiring an image; a means for determining whether or not to execute processing for detecting an object appearing in the image based on the size of an area in the image that shows a person; means for executing the process when the result of the determination indicates that the process should be executed; a means for identifying an area of ​​an object appearing in the image based on relative position information between the area and the person from the detection results of the processing; A device comprising:

Images