Judgment device, mobile unit, judgment method, and program

Abstract

Regardless of the work environment, it reduces processing load and achieves cost reduction. [Solution] Image data is acquired, a masking image is generated, feature quantities of the target object and objects are extracted from the masking image, each feature quantity is vectorized, the similarity between the feature quantities is obtained, and it is determined whether or not an object is the target object based on the similarity. In generating the masking image, a color space is selected, the processing area is transitioned, and the start and end points in the color space of the target object and objects are automatically identified based on the decrease in the number of pixels or the change in variance between adjacent areas, the color range of the target object and objects is extracted, and a masking process is applied to color ranges different from the color range of the target object and objects to generate a masking image.

Description

Technical Field

[0001] The present disclosure relates to a determination device, a moving body, a determination method, and a program.

Background Art

[0002] In recent years, in order to cope with the labor shortage accompanying future population decline, movements to apply digital technology, robot technology, etc. to various fields have been active. Also, in response to such movements, recently, movements to integrate AI (artificial intelligence), which has made remarkable progress, into digital technology, robot technology, etc. have also been active.

[0003] On the other hand, when replacing the work that has been done by humans with robots or the like, it is important to distinguish between the work object and the non-work object.

[0004] In order to meet the above requirements, there is disclosed an image processing device that determines the state of an object using an image, is mounted on a moving body, includes a camera that acquires a captured image in which the object is captured, and an image generation unit that generates a normal state image indicating a normal state based on the captured image acquired by the camera using an image generation model that has learned the normal state, and a determination unit that determines the state of the object. The determination unit calculates the similarity between the captured image and the normal state image, and determines that the object is abnormal when the calculated similarity is small (for example, see Patent Document 1).

[0005] There is also disclosed an image processing device that assumes outdoor work and includes a memory that stores a template image group including a plurality of images of a detection target, an acquisition unit that acquires an entire image including a collation target, and a control unit that acquires a representative image based on a part of the template image group and performs a similarity determination between at least a part of the entire image and the representative image (for example, see Patent Document 2).

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-133306 [Patent Document 2] Japanese Patent Publication No. 2024-133706 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, the technology described in Patent Document 1 above requires a huge amount of training data to generate the trained image generation model, which takes a lot of time and money. Furthermore, there was the challenge that expensive processors such as GPUs were indispensable for performing training and inference at high speed using vast amounts of training data.

[0008] Furthermore, while the technology described in Patent Document 2 above proposes high-speed image recognition by matching carefully selected pixels that are resistant to the effects of disturbances, it has the problem that it cannot handle cases where the image of the object changes due to changes in sunlight intensity, or where the object itself changes over time, or even if it can handle such cases, it would require preparing various learning models for each situation from an even larger amount of training data.

[0009] Therefore, this disclosure has been made in view of the above-mentioned problems, and aims to provide a determination device, a mobile device, a determination method, and a program that reduce processing load and lower costs regardless of the working environment. [Means for solving the problem]

[0010] Embodiment 1; One or more embodiments of the present disclosure include an image data acquisition unit that acquires reference image data of a reference object from captured image data, and also acquires identification target image data of an object to be identified, and for the reference image data acquired by the image data acquisition unit, a color space is selected, processing areas are transitioned, and the start and end points of the reference object in the color space are automatically identified based on the decrease in the number of pixels or the change in variance between adjacent areas, and the reference object standard Extract the color range, and the aforementioned reference object of the extracted reference object standard Apply masking to a color range different from the color range. standard A masking image is generated, and for the identification target image data acquired by the image data acquisition unit, the reference image data is used. Extracted The aforementioned standard Based on the color range, the object to be identified standard Apply masking to a color range different from the color range. Identification target A masking image generation unit that generates a masking image, and the masking process The aforementioned criteria Masking image and within the masked image to be identified The proposed determination device includes: a feature extraction unit that extracts predetermined feature quantities from the reference image data and the image data to be identified; an information processing unit that vectorizes the extracted feature quantities from the reference image data and the image data to be identified; a similarity acquisition unit that obtains the similarity between a vector based on the feature quantities of the reference image data and a vector based on the feature quantities of the image data to be identified; and a determination unit that determines whether or not the object to be identified is the reference object based on the obtained similarity.

[0011] Embodiment 2; One or more embodiments of the present disclosure are a determination method in a determination device including an image data acquisition unit, a masking image generation unit, a feature extraction unit, an information processing unit, a similarity acquisition unit, and a determination unit, wherein the image data acquisition unit performs a first step of acquiring reference image data of a reference object from captured image data, and the masking image generation unit performs a first step of selecting a color space, transitioning processing regions, and automatically identifying the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent regions, and the reference object standard Extract the color range, and the aforementioned reference object of the extracted reference object standard Apply masking to a color range different from the color range. standard A second step of generating a masking image, and the feature extraction unit, the masked standard Masking image inside A third step of extracting predetermined feature quantities from the reference image data; a fourth step of the image data acquisition unit acquiring identification target image data of the object to be identified from the captured image data; and a masking image generation unit, with respect to the identification target image data acquired in the fourth step, using the reference image data. Extracted The aforementioned standard Based on the color range, the object to be identified standard Apply masking to a color range different from the color range. Identification target A fifth step involves generating a masking image, and the feature extraction unit then processes the masked image. Within the masked image to be identifiedA determination method is proposed, which includes a sixth step of extracting predetermined feature quantities from the image data to be identified; a seventh step of the information processing unit vectorizing the extracted feature quantities of the reference image data and the image data to be identified; an eighth step of the similarity acquisition unit acquiring the similarity between a vector based on the feature quantities of the reference image data and a vector based on the feature quantities of the image data to be identified; and a ninth step of the determination unit determining whether or not the object to be identified is the target object based on the acquired similarity.

[0012] Embodiment 3; One or more embodiments of the present disclosure are programs for causing a computer to execute a determination method in a determination device including an image data acquisition unit, a masking image generation unit, a feature extraction unit, an information processing unit, a similarity acquisition unit, and a determination unit, wherein the image data acquisition unit performs a first step of acquiring reference image data of a reference object from captured image data, and the masking image generation unit performs a first step of selecting a color space, transitioning processing regions, and automatically identifying the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent regions, and the reference object standard Extract the color range, and the aforementioned reference object of the extracted reference object standard Apply masking to a color range different from the color range. standard A second step of generating a masking image, and the feature extraction unit, the masked standard Masking image inside A third step of extracting predetermined feature quantities from the reference image data; a fourth step of the image data acquisition unit acquiring identification target image data of the object to be identified from the captured image data; and a masking image generation unit, with respect to the identification target image data acquired in the fourth step, using the reference image data. Extracted The aforementioned standard Based on the color range, the object to be identified standardPerforming masking processing on a color range different from the color range Identification target A fifth step of generating a masking image, and the feature extraction unit extracts predetermined features from the masked Within the masked image to be identified A sixth step of extracting respective predetermined feature amounts from the identification target image data, a seventh step of the information processing unit vectorizing the respective feature amounts of the extracted reference image data and the identification target image data, an eighth step of the similarity acquisition unit acquiring a similarity between a vector based on the feature amount of the reference image data and a vector based on the feature amount of the identification target image data, and a ninth step of the determination unit determining whether or not the object to be identified is the target object based on the acquired similarity. A program for causing a computer to execute the determination method is proposed.

[0013] Embodiment 4; One or more embodiments of the present disclosure acquire reference image data of a reference object from captured image data, and an image data acquisition unit that acquires identification target image data of an object to be identified. For the reference image data acquired by the image data acquisition unit, a color space is selected, a processing region is transitioned, and based on the decrease amount of the number of pixels or the change amount of the variance between adjacent regions at that time, the start point and the end point in the color space of the reference object are automatically specified, and the color range of the reference object is extracted. Performing masking processing on a color range different from the extracted color range of the reference object to generate a masking image. For the identification target image data acquired by the image data acquisition unit, based on the color range in the reference image data, masking processing is performed on a color range different from the color range of the object to be identified to generate a masking image, a masking image generation unit, and the masked standard masking image standard Performing masking processing on a color range different from the color range standard Generating a masking image, and for the identification target image data acquired by the image data acquisition unit, based on the color range in the reference image data Extracted the standard Based on the color range, for a color range different from the color range of the object to be identified standard Performing masking processing Identification target To generate a masking image, and the masked The aforementioned criteria masking image and within the masked image to be identifiedThe proposed mobile body includes: a feature extraction unit that extracts predetermined feature quantities from the reference image data and the image data to be identified; an information processing unit that vectorizes the extracted feature quantities from the reference image data and the image data to be identified; a similarity acquisition unit that obtains the similarity between a vector based on the feature quantities of the reference image data and a vector based on the feature quantities of the image data to be identified; and a determination unit that determines whether or not the object to be identified is the reference object based on the obtained similarity. [Effects of the Invention]

[0014] According to one or more embodiments of this disclosure, processing load can be reduced and costs can be lowered regardless of the working environment. [Brief explanation of the drawing]

[0015] [Figure 1] This figure shows the configuration of the determination device according to the present disclosure. [Figure 2] This figure illustrates the process of acquiring a reference image or an image to be identified in the determination device according to the embodiment of this disclosure. [Figure 3] This figure illustrates the background image masking process in a determination device according to an embodiment of the present disclosure. [Figure 4] This figure shows the decrease in the number of pixels or the change in variance within the color range of the reference image and the image to be identified in the determination device according to the embodiment of this disclosure. [Figure 5] This figure illustrates the feature extraction process in a determination device according to an embodiment of the present disclosure. [Figure 6] This figure illustrates the feature extraction process in a determination device according to an embodiment of the present disclosure. [Figure 7] This figure illustrates the feature extraction process in a determination device according to an embodiment of the present disclosure. [Figure 8] This figure illustrates the determination process in a determination device according to an embodiment of the present disclosure. [Figure 9] This figure shows the processing flow in the determination device according to the embodiment of this disclosure. [Figure 10] This figure shows the verification results of the determination device according to the embodiment of this disclosure. [Figure 11] This figure shows the verification results of the determination device according to the embodiment of this disclosure. [Modes for carrying out the invention]

[0016] <Embodiment> The determination device 1 according to this embodiment will be described with reference to Figures 1 to 11. In the following, the determination device 1 according to this embodiment will be specifically described using the example of a device mounted on a robot that performs a process of identifying crops from among the furrows of a field. However, the determination device 1 according to this embodiment is not limited to a device mounted on a robot that performs the process of identifying crops from among the furrows of a field. For example, it could be a device mounted on a robot that identifies weeds in the furrows of a field and removes them.

[0017] <Configuration of the determination device 1> As shown in Figure 1, the determination device 1 according to this embodiment is configured to include a processor 100 that performs processing, a storage unit 200, and a camera 300 that serves as an imaging unit.

[0018] Camera 300 acquires image data, for example, that includes the furrows in a field. Camera 300 captures images including the furrows of the field as the robot moves, for example, as shown in Figure 2. Camera 300 captures images as the robot moves, for example, as shown in Figure 2, including images of crops as the target object, and images of weeds (object 1) and crops (object 2) as the objects to be identified. There are no particular restrictions on the resolution of the Camera 300. Therefore, the camera 300 may be an image sensor or the like. Furthermore, the image data captured by camera 300 may be either video or still images.

[0019] <Configuration of Processor 100> As shown in Figure 1, the processor 100 is configured to include an image data acquisition unit 110, a masking image generation unit 120, a feature extraction unit 130, an information processing unit 140, a similarity acquisition unit 150, a determination unit 160, and a control unit 170.

[0020] The image data acquisition unit 110 acquires reference image data of a reference object (e.g., crops) from the image data captured by the camera 300, and also acquires identification target image data of object 1 (e.g., weeds) and object 2 (crops) to be identified. Note that the reference image data may be obtained before the image data to be identified is obtained. Alternatively, previously acquired image data may be selected and used as reference image data if the environmental conditions, growth status, and other conditions that cause changes in the feature values, as well as the state of the object itself (e.g., crops), match to a certain extent. Furthermore, the image data acquisition unit 110 may acquire the reference image data or the image data to be identified in multiple steps. The reference image data and the image data to be identified acquired by the image data acquisition unit 110 are sent to the control unit 170, which will be described later, via the bus line BL. The control unit 170 temporarily stores the reference image data and the image data to be identified, which are input from the image data acquisition unit 110, in the storage unit 200, which will be described later.

[0021] The masking image generation unit 120, for reference image data acquired by the image data acquisition unit 110, selects a color space, transitions the processing area, and automatically identifies the start and end points in the color space of the reference object based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the color range of the reference object, and generates a masked image by applying masking processing to color ranges different from the extracted color range of the reference object. For identification target image data acquired by the image data acquisition unit 110, the masking image generation unit 120 generates a masked image by applying masking processing to color ranges different from the color range of the object to be identified, based on the color range defined in the reference image data. The masking image generated in the masking image generation unit 120 is sent via the bus line BL to the control unit 170, which will be described later.

[0022] The masking image generation unit 120 generates a masking image by applying masking processing to color regions that differ from the color information of the object (e.g., crop) to reference image data in which the object (e.g., crop) has been captured, for example as shown in Figure 3, so as to transition from the original image in Figure 3(A) to Figure 3(E). Specifically, the masking image generation unit 120 selects a color space for the reference image data acquired by the image data acquisition unit 110, transitions the processing area, and automatically identifies the start and end points in the color space of the reference object based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the color range of the reference object, and applies masking processing to the color range that is different from the extracted color range of the reference object to generate a masking image. Furthermore, as shown in Figure 3, the masking image generation unit 120 also generates a masking image for the object 1 (e.g., weeds) and object 2 (e.g., crops) to be identified by masking the image data of the object 1 (e.g., weeds) and object 2 (e.g., crops) to be identified by masking a color region different from the color information of the object 1 (e.g., weeds) and object 2 (e.g., crops) to be identified. Specifically, the masking image generation unit 120 applies a masking process to the identification target image data acquired by the image data acquisition unit 110, based on the color range defined in the reference image data, for a color range different from the color range of the object to be identified, thereby generating a masking image. In other words, the masking image generation unit 120 automatically applies a masking process to the image data to be identified acquired by the image data acquisition unit 110, using the color range defined in the reference image data, and masking a color range different from the color range of the object to be identified, thereby generating a masking image. Therefore, since the masking image generation process is performed in an extremely short time, even when used in a device mounted on a robot that performs a process of identifying crops from among the rows of a field, as in this embodiment, appropriate processing can be performed in a short time.

[0023] The masking image generation unit 120 selects a color space for the reference image data acquired by the image data acquisition unit 110, transitions the processing area, and automatically identifies the start and end points in the color space of the reference object based on the decrease in the number of pixels or the change in variance between adjacent areas, thereby extracting the color range of the reference object (for example, a crop). Here, Figure 4 shows the decrease in the number of pixels and the change in variance in each pixel region. As shown in Figure 4, at the boundary between the reference image data and the background, the number of pixels decreases sharply or the variance changes from decreasing to increasing. Therefore, the masking image generation unit 120, for example, transitions the processing area and, based on the decrease in the number of pixels or the change in the variance between adjacent areas at that time, automatically identifies the start and end points in the color space of the reference object (e.g., crop) and extracts the color range of the reference object (e.g., crop) (the "first color range" in Figure 4). As a result, the processing load for extracting the boundaries between the reference image data, the image data to be identified, and the background is reduced. Therefore, the determination device 1 according to this embodiment is particularly suitable as a determination device mounted on a mobile body that performs determinations while moving. Furthermore, while the first color range in Figure 4 shows the region where the number of pixels or variance changes, based on the characteristics of the variance change at the boundary between the reference image data and the background, the processing load can be further reduced by using a second color range (the "second color range" in Figure 4) that includes the point where the variance begins to decrease and reaches its minimum.

[0024] The masking image generation unit 120 promptly executes the masking image generation process after acquiring image data by the image data acquisition unit 110. In other words, after acquiring the reference image data, a masked image containing the reference image data is quickly generated, and after acquiring the image data to be identified, a masked image containing the image data to be identified is quickly generated. Therefore, the subsequent feature extraction process by the feature extraction unit 130, the feature vectorization process by the information processing unit 140, the similarity acquisition process by the similarity acquisition unit 150, and the determination process by the determination unit 160 are executed in a short amount of time. Therefore, the determination device 1 according to this embodiment is particularly suitable as a determination device mounted on a mobile body that performs determination while moving.

[0025] The masking image generation unit 120 may select, for example, the Lab color space as the color space. The Lab color space is close to human perception and has the characteristic of easily quantifying simple perceptions such as greenish or yellowish tones. Therefore, by selecting, for example, the Lab color space, it becomes possible to replace the work that was conventionally performed by a person visually making judgments with a mobile device such as a robot equipped with the judgment device 1 according to this embodiment. Furthermore, by using the judgment device 1 according to this embodiment, it is expected that judgment results equivalent to or better than those of human judgment can be achieved at a speed equivalent to or faster than human judgment, without the need for expensive machine learning. The masking image generated by the masking image generation unit 120 is sent via the bus line BL to the control unit 170, which will be described later. The control unit 170 temporarily stores each digitized feature in the storage unit 200, which will be described later.

[0026] The feature extraction unit 130 extracts predetermined features from the reference image data and the image data to be identified, respectively, from the masking image generated by the masking image generation unit 120. The feature extraction unit 130 extracts features related to color, shape, and size from the reference image data and the image data to be identified in the masking image generated by the masking image generation unit 120. The feature extraction unit 130 extracts features including the variation in pixel values ​​or the average value of hue for color, features including the maximum length of the contour line or the aspect ratio at the largest contour for shape, and features including the total number of extracted pixels or the maximum height for size.

[0027] The feature extraction unit 130 extracts numerically represented features based on a rule system from the reference image data and the image data to be identified (Figure 6(B)), which are obtained by extracting pixel information other than the mask color (black) from the mask image (Figure 6(A)) generated by the mask image generation unit 120, for example, as shown in Figures 5 to 7. In the example shown in Figure 5, examples of features include color, shape, and size. The feature extraction unit 130 extracts features such as the variation in pixel values ​​and the average value of hue obtained from the masking image for color. In this embodiment, for example, an example is shown in which the RGB values ​​of the RGB color space acquired by the image data acquisition unit 110 are used as pixel values, and their variation is extracted as a feature quantity, but the embodiment is not limited to this. Furthermore, in this embodiment, for example, an example is shown in which the hue of the RGB color space acquired by the image data acquisition unit 110 is converted to the hue of the HSV color space and the average value of the hue is calculated, but the embodiment is not limited to this. The feature extraction unit 130, for example, obtains contour information (from A to F in Figure 7(A)) from the masking image for shape, and extracts features such as the length of the contour line (for example, the length of the dotted line in E in Figure 7(A)) and the aspect ratio of the contour (for example, Y / X in E in Figure 7(A)). In this embodiment, for example, the length of the contour line and the aspect ratio of the contour were used as examples of features to be extracted. However, the features to be extracted will change depending on the reference object and the object to be identified. The feature extraction unit 130, for example, obtains all pixel information of a block of region having contour information (for example, in Figure 7(C)) from the masking image regarding size, and extracts feature quantities such as the total number of pixels and height information from the ground based on depth information. In this embodiment, for example, the total number of pixels and height information from the ground were used as examples of features to be extracted. However, the features to be extracted will change depending on the reference object and the object to be identified. Each feature, quantified by the feature extraction unit 130, is sent via the bus line BL to the control unit 170, which will be described later. The control unit 170 temporarily stores each digitized feature in the storage unit 200, which will be described later.

[0028] The information processing unit 140 vectorizes the features of the reference image data and the image data to be identified, respectively, that were extracted by the feature extraction unit 130. Specifically, the information processing unit 140 converts the color, shape, and size, which are digitized feature quantities, into three-dimensional vectors in a three-dimensional coordinate system of color, shape, and size, as shown in Figure 5, for example. The processing results from the information processing unit 140 are sent via the bus line BL to the control unit 170, which will be described later. The control unit 170 temporarily stores each vectorized feature in the storage unit 200, which will be described later.

[0029] The similarity acquisition unit 150 acquires the similarity between a 3D vector based on the features of the reference image data processed by the information processing unit 140 and a 3D vector based on the features of the image data to be identified. Specifically, the similarity acquisition unit 150 acquires similarity by comparing the 3D vector (vector X) of the reference image data and the 3D vector (vector Y) of the image data to be identified in terms of both direction and length, as shown in Figure 8, for example. The similarity acquisition unit 150 determines, for example, as shown in Figure 8(A), that the similarity between the two 3D vectors is low if the lengths of the 3D vector (vector X) of the reference image data and the 3D vector (vector Y) of the image data to be identified are the same, but the angle θ between the two 3D vectors differs by more than a predetermined angle difference. Furthermore, as shown in Figure 8(B), the similarity acquisition unit 150 determines that the similarity between the two 3D vectors is low if the angle difference θ between the 3D vector of the reference image data (vector X) and the 3D vector of the image data to be identified (vector Y) is within a predetermined angle range, but the difference in the lengths of the two 3D vectors is outside the predetermined range. Furthermore, as shown in Figure 8(C), the similarity acquisition unit 150 determines that the similarity between the two 3D vectors is high if the angle difference θ between the 3D vector of the reference image data (vector X) and the 3D vector of the image data to be identified (vector Y) is within a predetermined angle range, and the difference in the lengths of the two 3D vectors is within a predetermined range. The predetermined range for the angle θ between the two 3D vectors and the predetermined range for the lengths of the two 3D vectors may be changed as appropriate according to the specifications. In this embodiment, for example, if the angle difference θ between the 3D vector (vector X) of the reference image data of the reference object (crop) shown in Figure 2 and the 3D vector (vector Y) of the identification target image data of object 2 (crop) shown in Figure 2 is within a predetermined angle range, and the difference in the lengths of the two 3D vectors is within a predetermined range, then it is determined that the similarity between the two 3D vectors is high. The results obtained by the similarity acquisition unit 150 are sent to the control unit 170, which will be described later, via the bus line BL. The control unit 170 temporarily stores the similarity score obtained by the similarity acquisition unit 150 in the storage unit 200, which will be described later.

[0030] The determination unit 160 determines whether or not the object to be identified is the target object based on the similarity obtained by the similarity acquisition unit 150. Specifically, the determination unit 160 compares a predetermined threshold with the similarity obtained by the similarity acquisition unit 150 to determine whether or not the object to be identified is the target object. In this embodiment, the determination unit 160 determines that, for example, if the similarity acquisition unit 150 determines that the similarity between the 3D vector (vector X) of the reference image data of the reference object (crop) shown in Figure 2 and the 3D vector (vector Y) of the identification target image data of object 2 (crop) shown in Figure 2 is high, then object 2 shown in Figure 2 is determined to be a crop. The determination result from the determination unit 160 is sent to the control unit 170, which will be described later, via the bus line BL.

[0031] The control unit 170 controls the operation of the entire determination device 1 based on a control program stored in ROM (Read Only Memory) or the like. More specifically, the control unit 170 controls the image data acquisition process in the image data acquisition unit 110, the masking image generation process in the masking image generation unit 120, the feature extraction process in the feature extraction unit 130, the information processing in the information processing unit 140, the similarity acquisition process in the similarity acquisition unit 150, the determination process in the determination unit 160, and so on.

[0032] The memory unit 200 consists of ROM or RAM (Random Access Memory), and stores programs, data, etc. In this embodiment, the storage unit 200 stores image data acquired by the camera 300, masking images, feature information, vector information, similarity information, threshold information, and the like.

[0033] <Processing by the determination device 1> The processing in the determination device 1 according to this embodiment will be explained using Figure 9.

[0034] The image data acquisition unit 110 acquires reference image data of a reference object from the image data captured by the camera 300 (step S110). The reference image data acquired by the image data acquisition unit 110 is sent to the control unit 170, which will be described later, via the bus line BL.

[0035] The masking image generation unit 120 selects a color space for the reference image data acquired by the image data acquisition unit 110, transitions the processing area, and automatically identifies the start and end points in the color space of the reference object based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the color range of the reference object, and applies masking processing to color ranges different from the extracted color range of the reference object to generate a masking image (step S120). The masking image of the reference image generated by the masking image generation unit 120 is sent via the bus line BL to the control unit 170, which will be described later.

[0036] The feature extraction unit 130 extracts predetermined features from the reference image data in the masking image of the reference image data generated by the masking image generation unit 120 (step S130). More specifically, in this embodiment, examples of feature quantities include color, shape, size, etc. The feature quantity extraction unit 130 extracts, for example, feature quantities including the variation in pixel values ​​or the average value of hue for color, feature quantities including the maximum length of the contour line or the aspect ratio at the largest contour for shape, and feature quantities including the total number of extracted pixels or the maximum height for size. Each feature, quantified by the feature extraction unit 130, is sent via the bus line BL to the control unit 170, which will be described later.

[0037] The image data acquisition unit 110 acquires the image data to be identified from the image data captured by the camera 300 (step S140). The image data to be identified, acquired by the image data acquisition unit 110, is sent via the bus line BL to the control unit 170, which will be described later.

[0038] The masking image generation unit 120 applies a masking process to the identification target image data acquired by the image data acquisition unit 110, based on the color range defined in the reference image data, to a color range different from the color range of the object to be identified, thereby generating a masking image (step S150). The masking image of the reference image generated by the masking image generation unit 120 is sent via the bus line BL to the control unit 170, which will be described later.

[0039] The feature extraction unit 130 extracts predetermined features from the image data to be identified using the masking image of the image to be identified generated by the masking image generation unit 120 (step S160). More specifically, in this embodiment, examples of feature quantities include color, shape, size, etc. The feature quantity extraction unit 130 extracts, for example, feature quantities including the variation in pixel values ​​or the average value of hue for color, feature quantities including the maximum length of the contour line or the aspect ratio at the largest contour for shape, and feature quantities including the total number of extracted pixels or the maximum height for size. Each feature, quantified by the feature extraction unit 130, is sent via the bus line BL to the control unit 170, which will be described later.

[0040] In steps S130 and S160, the information processing unit 140 converts the feature quantities of the reference image data and the image data to be identified, extracted by the feature extraction unit 130, into three-dimensional vectors (step S170). Specifically, in this embodiment, the information processing unit 140 converts the color, shape, and size, which are digitized feature quantities, into three-dimensional vectors in a three-dimensional coordinate system of color, shape, and size. The processing results from the information processing unit 140 are sent via the bus line BL to the control unit 170, which will be described later.

[0041] The similarity acquisition unit 150 acquires the similarity between a 3D vector based on the features of the reference image data processed by the information processing unit 140 and a 3D vector based on the features of the image data to be identified (step S180). Specifically, the similarity acquisition unit 150 acquires similarity by comparing the 3D vector (vector X) of the reference image data and the 3D vector (vector Y) of the image data to be identified in terms of both direction and length, as shown in Figure 8, for example. The results obtained by the similarity acquisition unit 150 are sent to the control unit 170, which will be described later, via the bus line BL.

[0042] The determination unit 160 determines whether or not the object to be identified is the target object based on the similarity obtained by the similarity acquisition unit 150 (step S190). Specifically, the determination unit 160 compares a predetermined threshold with the similarity obtained by the similarity acquisition unit 150 to determine whether or not the object to be identified is the target object. The determination result from the determination unit 160 is sent to the control unit 170, which will be described later, via the bus line BL.

[0043] The control unit 170 determines whether all processing has been completed on the reference image data and the identification target image data of the object to be identified within the image data acquired by the image data acquisition unit 110 (step S191). Then, if the control unit 170 determines that all processing has been completed on the reference image data and the identification target image data of the object to be identified within the image data acquired by the image data acquisition unit 110 (YES in step S191), it terminates all processing.

[0044] On the other hand, if the control unit 170 determines that all processing of the reference image data and the identification target image data of the object to be identified within the image data acquired by the image data acquisition unit 110 is not yet complete (NO in step S191), it transitions the process to step S140.

[0045] <Verification Results> The actual verification results of the determination device 1 according to the embodiment described above will be explained using Figures 10 and 11. Figure 10 shows the results of verifying the accuracy of the identification method using okra, green onions, and carrots as the target objects (crops). Furthermore, for okra, the study was conducted at different times, in September and October, to examine the effects of changes in solar radiation conditions such as the angle of sunlight and changes in the growth process.

[0046] The verification results are shown in Figure 10, and in all cases the judgment accuracy was 95% or higher, indicating a favorable result. Furthermore, in this verification, the determination was made using image data with a resolution reduced to 1 / 16th of the normal resolution. In other words, this verification proved that even with low-resolution image data, sufficient detection accuracy is possible. Furthermore, the processing time for the judgment was 5ms, which was significantly shorter than the processing time set before the verification. Furthermore, in this verification, we investigated how the angle of sunlight, i.e., shadows and reflections, affects the accuracy of the determination of the same object (crop; okra), and proved that the difference in determination accuracy was within a negligible range.

[0047] Figure 11 shows the results of verifying whether it is possible to identify only capsules with the same shape as the reference image from among similar capsules with slightly different shapes. The verification results proved that accurate determination can be made by considering not only the color of the capsule as the object, but also its size and shape as determination factors.

[0048] <Effects and Actions> As described above, the determination device 1 according to this embodiment includes an image data acquisition unit 110 that acquires reference image data of a reference object from captured image data, and also acquires identification target image data of an object to be identified. For the reference image data acquired by the image data acquisition unit 110, a color space is selected, the processing area is transitioned, and based on the decrease in the number of pixels or the change in variance between adjacent areas at that time, the start and end points in the color space of the reference object are automatically identified to extract the color range of the reference object, and masking processing is applied to color ranges different from the extracted color range of the reference object to generate a masked image. The system includes: a masking image generation unit 120 that generates a masking image by applying a masking process to a color range different from the color range of the object to be identified, based on a color range defined in the reference image data; a feature extraction unit 130 that extracts predetermined feature quantities from the reference image data and the image data to be identified, respectively, in the masked image; an information processing unit 140 that vectorizes the extracted feature quantities of the reference image data and the image data to be identified; a similarity acquisition unit 150 that obtains the similarity between a vector based on the feature quantities of the reference image data and a vector based on the feature quantities of the image data to be identified; and a determination unit 160 that determines whether or not the object to be identified is the reference object based on the obtained similarity. In other words, in the determination device 1 according to this embodiment, determination is made based on the reference image data and the image data to be identified acquired by the image data acquisition unit 110. Therefore, when there are variable factors such as season, weather, time of day, or, if the target object is a crop, changes in the object itself (such as growth status), it was previously necessary to load a huge amount of data and perform high-speed processing to generate a learning model that corresponds to such variable factors. However, with this new system, the determination process can be performed using reference image data and image data of the object to be identified acquired at the work site. Therefore, since expensive processors are not required, cost reduction can be achieved regardless of the work environment. Furthermore, in the determination device 1 according to this embodiment, the masking image generation unit 120 selects a color space for the reference image data acquired by the image data acquisition unit 110, transitions the processing area, and automatically identifies the start and end points in the color space of the reference object based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the color range of the reference object, and generates a masking image by applying masking processing to a color range different from the extracted color range of the reference object. For the identification target image data acquired by the image data acquisition unit 110, the masking image generation unit 120 generates a masking image by applying masking processing to a color range different from the color range of the object to be identified, based on the color range defined in the reference image data. In other words, in the masking process for the image data to be identified, the color range defined in the reference image data is used as is, and the masking process is applied to a color range that is different from the color range of the object to be identified, thereby generating a masked image. This reduces the processing load required to detect the boundary between the reference image data, the image data to be identified, and the background. Furthermore, the determination device 1 according to this embodiment is particularly suitable as a determination device mounted on a mobile body that performs determinations while moving. Furthermore, at the boundary between the reference image data, the image data to be identified, and the background, the variance begins to decrease, and then tends to increase sharply. Therefore, taking this characteristic into account, if we define a new color range that includes the point where the variance begins to decrease and reaches its minimum, the processing load can be further reduced. Furthermore, in the determination device 1 according to this embodiment, predetermined feature quantities are extracted from the reference image data and the image data to be identified, respectively, the extracted feature quantities of the reference image data and the image data to be identified are vectorized, the similarity between the vector based on the feature quantities of the reference image data and the vector based on the feature quantities of the image data to be identified is obtained, and based on the obtained similarity, it is determined whether or not the object to be identified is the target object. In other words, masking allows for the efficient extraction of features by excluding background information, or unnecessary information. This improves the validity of the information and reduces the processing load. Therefore, it is possible to further reduce the processing load and achieve cost reductions regardless of the work environment. Therefore, the determination device 1 according to this embodiment offers the unique advantage of being able to replace the processing that was conventionally left to machine learning with a determination method that uses inexpensive hardware and has a low processing load.

[0049] In the determination device 1 according to this embodiment, the masking image generation unit 120 promptly performs the masking image generation process after acquiring image data by the image data acquisition unit 110. In other words, after acquiring the reference image data, a masked image containing the reference image data is quickly generated, and after acquiring the image data to be identified, a masked image containing the image data to be identified is quickly generated. Therefore, the subsequent feature extraction process by the feature extraction unit 130, the feature vectorization process by the information processing unit 140, the similarity acquisition process by the similarity acquisition unit 150, and the determination process by the determination unit 160 are executed promptly. Therefore, the determination device 1 according to this embodiment is particularly suitable as a determination device mounted on a mobile body that performs determinations while moving.

[0050] In the determination device 1 according to this embodiment, the masking image generation unit 120 selects the Lab color space as the color space. The Lab color space is close to human perception and has the characteristic of easily quantifying simple perceptions such as greenish or yellowish tones. Therefore, by selecting, for example, the Lab color space, it becomes possible to replace the work that was conventionally performed by a person visually making judgments with a mobile device such as a robot equipped with the judgment device 1 according to this embodiment. Furthermore, by using the judgment device 1 according to this embodiment, it is expected that judgment results equivalent to or better than those of human judgment can be achieved with a judgment speed equivalent to or better than that of a human, without the need for expensive machine learning processing.

[0051] In the determination device 1 according to this embodiment, the feature extraction unit 130 extracts features related to color, shape, and size from the reference image data and the image data to be identified in the masked image. Color, shape, and size are the three elements that humans use to perceive objects visually. Therefore, by using color, shape, and size as the target features for extraction, it becomes possible to replace the work that was conventionally performed by humans visually with a mobile device such as a robot equipped with the judgment device 1 according to this embodiment.

[0052] In the determination device 1 according to this embodiment, the feature extraction unit 130 extracts features including the variation in pixel values ​​or the average value of hue for color, features including the maximum length of the contour line or the aspect ratio at the largest contour for shape, and features including the total number of extracted pixels or the maximum height for size. In other words, by using features such as the variation in pixel values ​​or the average value of hue for color, the maximum length of contour lines or the aspect ratio at the largest contour for shape, and the total number of extracted pixels or the maximum height for size, it becomes possible to replace the work that was conventionally performed by humans visually with a mobile device such as a robot equipped with the judgment device 1 according to this embodiment.

[0053] In the determination device 1 according to this embodiment, the information processing unit 140 generates a three-dimensional vector with feature quantities related to color, shape, and size as the three axes. In other words, the information processing unit 140 generates a three-dimensional vector with digitized color, shape, and size features as its three axes, further compressing the object to be judged and thus enabling a further reduction in processing load.

[0054] In the determination device 1 according to this embodiment, the similarity acquisition unit 150 determines the similarity based on both the length and direction of a vector based on the feature quantities of the reference image data and a vector based on the feature quantities of the image data to be identified. If the determination process places a large emphasis on the direction of both 3D vectors, the determination device 1 according to this embodiment may not be able to obtain the expected determination accuracy. Therefore, the similarity acquisition unit 150 determines the similarity based on both the length and direction of the vector based on the features of the reference image data and the vector based on the features of the image data to be identified, thereby obtaining a determination result that is consistent with the original expected value.

[0055] <Example 1> In this embodiment, the process by which the feature extraction unit 130 extracts predetermined features from the reference image data and the image data to be identified acquired by the image data acquisition unit 110 was described as an example. However, if, at some point, the image data acquired by the image data acquisition unit 110 does not include, for example, the reference image data, the feature extraction process in the feature extraction unit 130 may be performed first on the acquired identification target image data, and then, when the reference image data is acquired, the feature extraction unit 130 may be performed on the reference image data.

[0056] <Modification 2> In this embodiment, we have described how to determine whether or not an object to be identified is an object based on image data. However, the method is not limited to this, and for example, audio data or the like may be used to determine whether or not an object to be identified is an object.

[0057] <Variation 3> In this embodiment, a configuration in which the similarity acquisition unit 150 and the determination unit 160 are provided separately has been described as an example. However, the determination unit 160 may also be configured to include the processing of the similarity acquisition unit 150, so that the determination unit 160 performs the acquisition and determination of similarity.

[0058] Furthermore, the determination device 1 of this disclosure can be realized by recording the processing of the processor 100 on a recording medium readable by a computer system, and then having the processor 100 read and execute the program recorded on this recording medium. The computer system referred to here includes hardware such as an operating system and peripheral devices.

[0059] Furthermore, "computer system" shall also include the homepage provisioning environment (or display environment) if the WWW (World Wide Web) system is being used. Furthermore, the above program may be transmitted from a computer system that stores this program in a memory device or the like to another computer system via a transmission medium or by transmission waves within the transmission medium. Here, the "transmission medium" used to transmit a program refers to a medium that has the function of transmitting information, such as a network (communication network) like the Internet or a communication line (communication line) like a telephone line.

[0060] Furthermore, the above program may be intended to implement some of the functions described above. Furthermore, the aforementioned functions may be realized in combination with programs already recorded in the computer system, such as so-called differential files (differential programs).

[0061] As described above with reference to the drawings, all determination devices that a person skilled in the art can implement by appropriately modifying the design based on the determination device 1 described above as an embodiment of this disclosure also fall within the technical scope of the present invention, insofar as they encompass the gist of this disclosure. Within the scope of the concept of this invention, a person skilled in the art can conceive of various modifications and alterations, and it is understood that these modifications and alterations also fall within the technical scope of this invention. For example, any embodiment described above that a person skilled in the art has modified by adding, deleting, or changing the design of components, or by adding, omitting, or changing the conditions of a process, is also included within the technical scope of the present invention, as long as it retains the essence of the present invention.

[0062] Furthermore, any other effects and advantages brought about by the embodiments described herein that are obvious from this specification or that can be appropriately conceived by those skilled in the art are naturally considered to be brought about by the present invention. Various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be removed from all the components shown in the embodiment. Furthermore, components from different embodiments may be combined as appropriate. [Explanation of Symbols]

[0063] 1; Judgment device 100; processor 110; Image data acquisition unit 120; Masking image generation unit 130; Feature extraction unit 140; Information Processing Unit 150;Similarity acquisition part 160;judgment section 170; Control Unit 200;Memory section 300; Imaging unit

Claims

1. An image data acquisition unit that acquires reference image data of a reference object from captured image data, and also acquires identification image data of an object to be identified, A masking image generation unit selects a color space for the reference image data acquired by the image data acquisition unit, transitions the processing area, and automatically identifies the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the reference color range of the reference object, applies masking to color ranges different from the extracted reference color range of the reference object to generate a reference masking image, and applies masking to color ranges different from the reference color range of the object to be identified based on the reference color range extracted in the reference image data to generate an identification target masking image, A feature extraction unit extracts predetermined feature quantities from the reference image data and the image data within the masked reference masking image and the masked image, respectively. An information processing unit that vectorizes the respective feature quantities of the extracted reference image data and the image data to be identified, A similarity acquisition unit that acquires the similarity between a vector based on the features of the reference image data and a vector based on the features of the image data to be identified, A determination unit that determines whether or not the object to be identified is the reference object based on the obtained similarity, A determination device, including a determination device.

2. The determination device according to claim 1, wherein the masking image generation unit promptly performs the generation process of the reference masking image and the identification target masking image after acquiring image data by the image data acquisition unit.

3. The determination device according to claim 2, wherein the masking image generation unit selects the Lab color space as the color space.

4. The determination device according to claim 3, wherein the feature extraction unit extracts the feature quantities relating to color, shape, and size from the reference image data and the image data to be identified within the masked reference masking image and the masking image to be identified.

5. The determination device according to claim 4, wherein the feature extraction unit extracts, for color, the feature including the variation in pixel values ​​or the average value of hue; for shape, the feature including the maximum length of the contour line or the aspect ratio at the maximum contour; and for size, the feature including the total number of extracted pixels or the maximum height.

6. The determination device according to claim 5, wherein the information processing unit generates a three-dimensional vector with the three axes being the feature quantities relating to the color, shape, and size.

7. The similarity acquisition unit determines the similarity based on the length and direction of the vector based on the feature quantities of the reference image data and the vector based on the feature quantities of the image data to be identified, according to claim 6.

8. A determination method in a determination device including an image data acquisition unit, a masking image generation unit, a feature extraction unit, an information processing unit, a similarity acquisition unit, and a determination unit, The image data acquisition unit performs a first step of acquiring reference image data of a reference object from the captured image data, The masking image generation unit performs a second step in which, with respect to the reference image data acquired in the first step, it selects a color space, transitions the processing area, and automatically identifies the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the reference color range of the reference object, and applies masking processing to a color range different from the extracted reference color range of the reference object to generate a reference masking image. The feature extraction unit performs a third step of extracting predetermined features from the reference image data within the masked reference masking image, The image data acquisition unit performs a fourth step of acquiring image data of the object to be identified from the captured image data, The masking image generation unit performs a masking process on the identification target image data acquired in the fourth step, based on the reference color range extracted in the reference image data, to a color range different from the reference color range of the object to be identified, thereby generating an identification target masking image. The feature extraction unit performs a sixth step of extracting predetermined features from the image data to be identified within the masked image to be identified, The information processing unit performs a seventh step of vectorizing the respective feature quantities of the extracted reference image data and the image data to be identified, The similarity acquisition unit performs an eighth step of acquiring the similarity between a vector based on the features of the reference image data and a vector based on the features of the image data to be identified, The determination unit performs a ninth step in which it determines whether or not the object to be identified is the target object based on the obtained similarity score, A determination method that includes this.

9. A program for causing a computer to execute a determination method in a determination device including an image data acquisition unit, a masking image generation unit, a feature extraction unit, an information processing unit, a similarity acquisition unit, and a determination unit, The image data acquisition unit performs a first step of acquiring reference image data of a reference object from the captured image data, The masking image generation unit performs a second step in which, with respect to the reference image data acquired in the first step, it selects a color space, transitions the processing area, and automatically identifies the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the reference color range of the reference object, and applies masking processing to a color range different from the extracted reference color range of the reference object to generate a reference masking image. The feature extraction unit performs a third step of extracting predetermined features from the reference image data within the masked reference masking image, The image data acquisition unit performs a fourth step of acquiring image data of the object to be identified from the captured image data, The masking image generation unit performs a masking process on the identification target image data acquired in the fourth step, based on the reference color range extracted in the reference image data, to a color range different from the reference color range of the object to be identified, thereby generating an identification target masking image. The feature extraction unit performs a sixth step of extracting predetermined features from the image data to be identified within the masked image to be identified, The information processing unit performs a seventh step of vectorizing the respective feature quantities of the extracted reference image data and the image data to be identified, The similarity acquisition unit performs an eighth step of acquiring the similarity between a vector based on the features of the reference image data and a vector based on the features of the image data to be identified, The determination unit performs a ninth step in which it determines whether or not the object to be identified is the target object based on the obtained similarity score, A program that includes a method for making a judgment, which is used to cause a computer to execute that method.

10. An image data acquisition unit that acquires reference image data of a reference object from captured image data, and also acquires identification image data of an object to be identified, A masking image generation unit selects a color space for the reference image data acquired by the image data acquisition unit, transitions the processing area, and automatically identifies the start and end points of the reference object in the color space based on the decrease in the number of pixels or the change in variance between adjacent areas, extracts the reference color range of the reference object, applies masking to color ranges different from the extracted reference color range of the reference object to generate a reference masking image, and applies masking to color ranges different from the reference color range of the object to be identified based on the reference color range extracted in the reference image data to generate an identification target masking image, A feature extraction unit extracts predetermined feature quantities from the reference image data and the image data within the masked reference masking image and the masked image, respectively. An information processing unit that vectorizes the respective feature quantities of the extracted reference image data and the image data to be identified, A similarity acquisition unit that acquires the similarity between a vector based on the features of the reference image data and a vector based on the features of the image data to be identified, A determination unit that determines whether or not the object to be identified is the reference object based on the obtained similarity, A mobile entity, including a mobile body.

Images