Processing apparatus, processing program, and processing method

Abstract

To provide a processing apparatus, a processing program, and a processing method which has higher reliability.SOLUTION: The present invention is directed to a processing apparatus having at least one processor. The at least one processor is configured to: acquire, through a communication interface, a subject image through an imaging device configured to capture one or a plurality of objects as a subject; acquire, based on a user operation input, an evaluation item as an object of evaluation in the acquired subject image; utilize a learned model configured to evaluate the object in the evaluation items based on the acquired subject image to evaluate the evaluation items based on the subject image; and carry out processing for outputting a result of evaluation as to the evaluation items.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a processing device, a processing program, and a processing method configured to execute processing related to the evaluation of a subject image.

Background Art

[0002] Conventionally, a technique for photographing a subject image of a subject and determining a predicted disease has been known. Patent Document 1 discloses "a work management system for evaluating the cleaning status of a work area to be cleaned where cleaning work is performed, comprising a portable information terminal having a photographing function possessed by a cleaning worker, and a work management device communicating with the portable information terminal via the Internet. The portable information terminal includes a recording unit for recording predetermined cleaning work points for the work area to be cleaned, a cleaning work execution unit for displaying the cleaning work points recorded in the recording unit on a screen, a photographing instruction unit for instructing photographing of the situation before and after cleaning with respect to the cleaning work points, and an image transmission unit for transmitting the photographed image information taken in accordance with the instruction of the photographing instruction unit to the management device. The management device includes a work site database in which photographing image information before and after cleaning at the cleaning work points is recorded, and a work evaluation unit for evaluating the cleaning work at the cleaning work points based on the photographing image information before and after cleaning recorded in the work site database, and a work management system for transmitting the evaluation result of the work evaluation unit to the portable information terminal." However, in this technique, the evaluation is simply made based on the situation at the time of photographing.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, based on the above technologies, an object of the present disclosure is to provide a processing device, a processing program, and a processing method that enable more reliable evaluation through various embodiments.

Means for Solving the Problems

[0005] According to one aspect of the present disclosure, there is provided "a processing device including at least one processor, wherein the at least one processor acquires a subject image via an imaging device configured to image one or more objects as subjects via a communication interface, acquires an evaluation item to be evaluated in the acquired subject image based on a user's operation input, and uses a learned model configured to evaluate the evaluation item based on the acquired subject image to evaluate the evaluation item based on the subject image, and executes a process for outputting a result of the evaluation regarding the evaluation item."

[0006] According to one aspect of the present disclosure, there is provided "a processing program that, when executed by at least one processor, causes the at least one processor to acquire a subject image via an imaging device configured to image one or more objects as subjects via a communication interface, acquire an evaluation item to be evaluated in the acquired subject image based on a user's operation input, use a learned model configured to evaluate the evaluation item based on the acquired subject image to evaluate the evaluation item based on the subject image, and function the at least one processor to output a result of the evaluation regarding the evaluation item."

[0007] According to one aspect of the present disclosure, there is provided "a processing method executed by at least one processor, the method including: obtaining a subject image via an imaging device configured to image one or more objects as a subject via a communication interface; obtaining an evaluation item to be evaluated in the obtained subject image based on a user operation input; evaluating the evaluation item based on the obtained subject image using a learned model configured to evaluate the evaluation item based on the obtained subject image; and outputting a result of the evaluation of the evaluation item."

Effects of the Invention

[0008] According to the present disclosure, various embodiments can provide a processing device, a processing program, and a processing method capable of more reliable evaluation.

[0009] Note that the above effects are merely exemplary for convenience of explanation and are not limiting. In addition to or instead of the above effects, any effects described in the present disclosure or effects obvious to those skilled in the art can also be achieved.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0011] 1. Overview of Processing System 1 The processing system 1 according to the present disclosure is mainly used to acquire a subject image from an imaging device configured to image one or more objects as a subject, evaluate an evaluation item based on the subject image, and acquire the result of the evaluation. In particular, the processing system 1 acquires an arbitrary evaluation item based on a user's operation input. Further, the processing system 1 evaluates an arbitrary evaluation item acquired using a learned model based on the subject image. Further, in the evaluation, the processing system 1 uses a value (quantitative value) quantified for the evaluation item and information (time-series information) in which the quantitative values are arranged in time series.

[0012] That is, such a processing system 1 can capture a temporal change in the quantitative value for an arbitrary evaluation item that the user focuses on in the subject image. When evaluating based only on an event that occurs instantaneously at a certain point in time from a subject image at a certain point in time, abnormalities caused by mere noise or temporary events are considered excessively. However, in the processing system 1, since the temporal change in the quantitative value is captured as described above, a more reliable evaluation is possible.

[0013] Such a processing system 1 can be suitably used, for example, at a work site in the primary or secondary industry. At such a work site, various articles (such as tools) are used and managed. The processing system 1 evaluates the appropriateness of the management situation from a subject image obtained by photographing such a work site as a subject. Such appropriateness of the management situation is known, for example, by the standards (such as ISO9000) defined by the International Organization for Standardization (ISO). Therefore, the processing system 1 can be particularly suitably used at a work site in the secondary industry to which such standards are applied. At such a work site, the processing system 1 at least photographs a subject image including a predetermined storage location of an article (such as a tool) as a subject, and can evaluate shortages of tools, a messy storage situation, etc. from the subject image in time series and notify the occurrence of an abnormality.

[0014] Note that the uses in the primary and secondary industries are merely examples of the uses of the processing system 1, and of course, it can also be used for other uses. For example, the processing system 1 can be used for various purposes such as evaluating the cleaning and tidying status of each room in a house, evaluating the situation on shelves where products are displayed in the retail industry or in a store, evaluating the management situation of articles stored in a warehouse, evaluating the intrusion situation of suspicious persons. In the following, for the sake of convenience of explanation, a case of evaluating the management situation of tool articles with a work site in the secondary industry as a subject will be taken as an example, but of course, the invention according to the present disclosure is not limited to this alone.

[0015] 2. Configuration of Processing System 1 FIG. 1 is a block diagram showing the configuration of a processing system 1 according to an embodiment of the present disclosure. According to FIG. 1, the processing system 1 includes a server device 100, a photographing device 200, and a terminal device 300, and each device is communicably connected via a wired or wireless network. The server device 100 evaluates, using a learned model, an evaluation item specified by a user's operation input in the terminal device 300 based on a subject image acquired from the photographing device 200, and outputs the result. The photographing device 200 photographs a subject image of a space (for example, a work site) including one or more articles as a subject. The terminal device 300 inputs an evaluation item to be evaluated by the server device 100, and receives and displays on a display or the like the result evaluated by the server device 100.

[0016] In the present disclosure, the processing device means the server device 100, the photographing device 200, the terminal device 300, or a combination thereof. That is, in the following, the case where the server device 100 functions as a processing device will be described, but the photographing device 200 and the terminal device 300 can also function as a processing device in the same manner. Further, in the present disclosure, the storage and processing performed by the processing device may be distributed to other terminal devices, other server devices, or the like. That is, the processing device is not limited to being configured from a single housing, and includes the server device 100, the photographing device 200, the terminal device 300, other terminal devices, other server devices, or a combination thereof.

[0017] Also, in FIG. 1, only one terminal device 300 and one photographing device 200 are shown. However, of course, it is also possible to manage and operate a plurality of terminal devices 300 or a plurality of photographing devices 200 according to the use, the number of users, and the like.

[0018] FIG. 2A is a block diagram showing the configuration of the server device 100 according to an embodiment of the present disclosure. According to FIG. 2A, the server device 100 includes a processor 111, a memory 112, and a communication interface 113. These components are electrically connected to each other via control lines and data lines. The server device 100 does not necessarily include all the components shown in FIG. 2A, and it is possible to configure it by omitting some of them, or to add other components. For example, it is also possible to use an external memory communicably connected as a memory, a database device, a server device, etc. Also, it is possible to distribute and execute some of the processing to a processing device including other server devices. That is, the server device 100 is not limited to a single device only, and includes cases where it is distributed among a plurality of devices according to the handling of its information and the processing load.

[0019] The processor 111 functions as a control unit that controls other components of the processing system 1 based on the processing program stored in the memory 112. The processor 111 executes processing for evaluating an evaluation item based on a captured image mainly using a learned model based on the processing program stored in the memory 112. Specifically, "processing for acquiring a subject image via an imaging device 200 configured to capture one or more objects as a subject via the communication interface 113", "processing for acquiring an evaluation item to be evaluated in the acquired subject image based on a user's operation input", "processing for evaluating an evaluation item based on the acquired subject image using a learned model configured to evaluate an evaluation item based on the subject image", and "processing for outputting the result of the evaluation regarding the evaluation item" are executed based on the processing program stored in the memory 112. The processor 111 is mainly composed of one or more CPUs, but may be combined with a GPU, an FPGA, etc. as appropriate.

[0020] The memory 112 is composed of a RAM, a ROM, a non-volatile memory, an HDD, an SSD, etc., and functions as a storage unit. The memory 112 stores, as a processing program, instruction commands for various controls of the processing system 1 according to this embodiment. Specifically, the memory 112 stores programs for the processor 111 to execute, such as "a process of acquiring a subject image via an imaging device 200 configured to capture one or more objects as a subject via a communication interface 113", "a process of acquiring evaluation items to be evaluated in the acquired subject image based on a user's operation input", "a process of evaluating the evaluation items based on the acquired subject image using a learned model configured to evaluate the evaluation items based on the subject image", and "a process of outputting the result of the evaluation regarding the evaluation items". In addition to the program, the memory 112 stores various information stored in an evaluation item management table and the like. Note that these information do not necessarily have to be always stored in the memory 112 in the server device 100, and may be stored in a remotely installed database device. In that case, the database device is also included in the memory 112.

[0021] The communication interface 113 functions as a notification unit for transmitting and receiving various information to and from an imaging device 200 and a terminal device 300 connected via a wired or wireless network. Examples of the communication interface 113 include various things such as a wired communication connector such as USB and SCSI, a broadband wireless communication such as a wireless LAN, Bluetooth (registered trademark), and LTE, a wireless communication transmission / reception device such as infrared rays, and various connection terminals for a printed mounting board and a flexible mounting board. The communication interface 113, for example, receives a subject image from the imaging device 200 or transmits an evaluation result to the terminal device 300.

[0022] FIG. 2B is a block diagram showing the configuration of the terminal device 300 according to an embodiment of the present disclosure. According to FIG. 2B, the terminal device 300 includes a processor 311, a memory 312, an input interface 313, an output interface 314, and a communication interface 315. These components are electrically connected to each other via control lines and data lines. Note that the terminal device 300 does not necessarily include all of the components shown in FIG. 2B, and may be configured by omitting some of them, or other components may be added. The terminal device 300 may be any device that can communicate with the server device 100 and the imaging device 200 via a wired or wireless network, and a smartphone device, a tablet device, a laptop PC device, a desktop PC device, an imaging device, etc. may be used as the terminal device 300.

[0023] The processor 311 functions as a control unit that controls other components of the processing system 1 based on the processing program stored in the memory 312. The processor 311 receives, from the user, an input of an evaluation item to be evaluated in the subject image based on the processing program stored in the memory 312, and transmits the input evaluation item to the server device 100. Further, the processor 311 receives, from the server device 100, the result of evaluating the subject image based on the processing program stored in the memory 312, and outputs the result via a display or the like. Specifically, the processor 311 executes, based on the processing program stored in the memory 312, "processing for receiving an operation input by the user for inputting an evaluation item to be evaluated in the subject image via the input interface 313", "processing for transmitting the evaluation item to be evaluated in the subject image to the server device 100 via the communication interface 315", "processing for receiving, from the server device 100, the result of evaluating the evaluation item based on the subject image via the communication interface 315", and "processing for outputting the received result to a display or the like". The processor 111 is mainly composed of one or more CPUs, but a GPU, an FPGA, etc. may be combined as appropriate.

[0024] The memory 312 is composed of a RAM, a ROM, a non-volatile memory, an HDD, an SSD, etc., and functions as a storage unit. The memory 312 stores, as a processing program, instruction commands for various controls of the processing system 1 according to the present embodiment. Specifically, the memory 312 stores a processing program for the processor 311 to execute, such as "processing for receiving an operation input by a user for inputting an evaluation item to be evaluated in a subject image via the input interface 313", "processing for transmitting an evaluation item to be evaluated in a subject image to the server device 100 via the communication interface 315", "processing for receiving, from the server device 100, a result of evaluating an evaluation item based on a subject image via the communication interface 315", and "processing for outputting the received result to a display or the like".

[0025] The input interface 313 functions as an input unit for receiving an operation input of a user to the terminal device 300. Examples of the input interface 313 include a physical key button and a touch panel having an input coordinate system corresponding to the display coordinate system of the display. In the case of a touch panel, an icon is displayed on the display, and by the operator performing an instruction input via the touch panel, a selection for each icon is made. The detection method for the instruction input of the subject by the touch panel may be any method such as a capacitance type or a resistive film type. The input interface 313 does not always need to be physically provided in the terminal device 300, and may be connected as needed via a wired or wireless network. Therefore, in addition to the above, a mouse, a keyboard, etc. can also be used as the input interface 313.

[0026] The output interface 314 functions as an output unit for outputting information such as the result of evaluation based on the subject image received from the server device 100. As an example of the output interface 314, a display composed of a liquid crystal panel, an organic EL display, a plasma display, or the like can be mentioned. However, it is not necessarily required that the terminal device 300 itself be equipped with a display. For example, an interface for connecting to a display or the like that can be connected to the terminal device 300 via a wired or wireless network can also function as the output interface 314 that outputs display data to the display or the like.

[0027] The communication interface 315 functions as a communication unit for transmitting and receiving various information between the server device 100 and the imaging device 200 connected via a wired or wireless network. As an example of the communication interface 315, various devices such as a wired communication connector such as USB or SCSI, a broadband wireless communication such as wireless LAN, Bluetooth (registered trademark), LTE, a wireless communication transceiver device such as infrared rays, and various connection terminals for a printed mounting board or a flexible mounting board can be mentioned.

[0028] Although not particularly shown in FIGS. 2A and 2B, the imaging device 200 includes a camera for capturing a subject image equipped with a sensor such as a CMOS, a communication interface for communicating with the server device 100, the terminal device 300, etc., an input interface for receiving a user's operation input, an output interface for outputting a through image or a captured image or the result of a user's operation input, a memory for storing a program executed by a processor and a captured subject image, and a processor for controlling these respective elements. These respective components are electrically connected to each other via a control line and a data line.

[0029] Also, the imaging device 200 having a camera is an example for acquiring a subject image. The imaging device 200 may include not only the imaging device 200 having a camera but also a detection device that detects a signal from a subject (for example, an infrared sensor, an ultraviolet sensor, an X-ray, a CT, an MRI, etc.).

[0030] 3. Various Information Used in the Processing in Processing System 1 FIG. 3 shows a table stored in the server device 100 and storing information provided to the imaging device 200 and the terminal device 300 according to the progress of the processing. These information are updated and stored as needed according to the progress of the processing. Each information shown in FIG. 3 may be stored in the memory 112 of the server device 100, or may be stored in another database device installed remotely and read out as needed according to the progress of the processing.

[0031] FIG. 3 is a diagram conceptually showing an evaluation item management table stored in the server device 100 according to an embodiment of the present disclosure. According to FIG. 3, in the evaluation item management table, input phrase information, opposite phrase information, feature information, subject image information, quantitative value information, threshold information, evaluation information, etc. are stored in association with the evaluation item ID information. The "evaluation item ID information" is information generated by the processor 111 each time a new evaluation item input by the user is acquired, and is unique information for identifying each evaluation item.

[0032] "Input phrase information" is information indicating a phrase that is a character string input as an evaluation item. Examples of such input phrase information include "room dirt", "dirt on the desk", "appropriateness of the storage location of items", "ripeness of harvested products", "degree of cherry blossom blooming", or combinations thereof. Thus, the input phrase information, which is an evaluation item, is information indicating a perspective for evaluating one or more objects included therein in the subject image. More specifically, examples of the input phrase information include phrases indicating the appropriateness of the management status of one or more items. Even more specifically, examples of the input phrase information include phrases indicating the appropriateness of the management status of one or more items at a work site in the primary industry or the secondary industry ("room dirt", "dirt on the desk", "appropriateness of the storage location of items", "ripeness of harvested products", or combinations thereof, etc.). Particularly specifically, examples of the input phrase information include phrases indicating the degree of disorder such as the storage location and orientation of items used at a work site in the primary industry or the secondary industry. Note that the input phrase information can include any of the phrases themselves input by receiving a user's operation input via the input interface 313 in the terminal device 300, phrases generated in the processing process after the phrases are acquired in the server device 100, and phrases finally converted into the form of a prompt (question sentence) by the processing.

[0033] "Opposite phrase information" is a phrase that indicates a string having an opposite meaning generated in the server device 100 with respect to an input phrase that is the input phrase information. Such a phrase is generated, for example, by inputting a prompt (question sentence) for generating a phrase having an opposite meaning based on the input phrase information to a large language model installed in the server device 100 or another device. Examples of such opposite phrase information include "cleanliness of the room", "cleanliness on the desk", "inappropriateness of the storage position of articles", "immaturity of harvested products", "degree of non-blooming of cherry blossoms", or combinations thereof. Note that the opposite phrase information can include any of the phrase itself having an opposite meaning generated based on the input phrase information, a phrase generated in the processing process after the phrase is generated in the server device 100, and a phrase finally converted into the form of a prompt (for example, a question sentence such as "What is a phrase having an opposite meaning to the phrase 'dirtiness of the room'?").

[0034] "Feature information" is information indicating visual features that represent each of the phrases input as input phrase information and the phrases generated as opposite phrase information, that is, an index that is focused on when evaluating the degree of each evaluation item indicated by each phrase. Such feature information is, for example, generated by inputting a prompt (for example, a question text such as "What are the visual features representing the phrase 'dirtiness of the room'?") for generating information indicating the visual features representing the phrase to a large language model installed in the server device 100 or another device. Examples of such feature information include, as visual features representing the phrase "dirtiness of the room" (input phrase information), "clothes are placed irregularly in the room", "the orientation of the books in the room is chaotic", "the ratio of the area where the floor surface of the room can be seen", or a combination thereof. Further, examples of such feature information include, as visual features representing, for example, "degree of non-blooming of cherry blossoms" (opposite phrase information), "the color of the petals is light", "the petals are partially blooming", "the petals are sparse", or a combination thereof.

[0035] "Subject image information" is image data of a subject image including one or more objects photographed using a photographing device 200 or the like as the subject. The image data may be any of one or more still images, one or more moving images, and combinations thereof. The subject image information is stored by being received from the photographing device 200 via the communication interface 113. The subject image information may be the image data itself photographed by the photographing device 200, or may be data after performing image processing such as sharpening on the image data. The subject image information is typically an image including a work site in the primary or secondary industry and articles used at the work site as the subject. The subject image information includes images photographed a plurality of times over an arbitrary period and information indicating the time of photographing in order to evaluate temporal changes regarding evaluation items of the subject. Here, the terms "photographing" and "subject image" are used for the purpose of explanation on the premise of the photographing device 200 having a camera. However, for example, in a detection device such as an infrared sensor, it is also possible to photograph a subject in the same manner and acquire a subject image (infrared image).

[0036] "Quantitative value information" is information indicating the result of evaluation in evaluation items estimated by inputting "input phrase information", "opposite phrase information", "feature information", and "subject image information" into a learned model for each of a plurality of subject images taken multiple times over an arbitrary period, and is information showing the result as a quantitative value. Conceptually, the quantitative value information is, as an example, the result of analyzing the subject image for each index generated as the feature information, digitizing them, and summing them up (for example, weighted average). Specifically, for example, the quantitative value information includes the visual features corresponding to the subject image to be evaluated and the input phrase information, the visual features corresponding to the subject image and the opposite phrase information, the similarity between the subject image and the learning correct image, and the similarity between the subject image and the learning incorrect image, and is a numerical value obtained by inputting them through a softmax function. The details of the process will be described later. Such quantitative value information is generated for each of the subject images taken multiple times over an arbitrary period as described above. Therefore, the quantitative value information includes information indicating the time when the subject image was taken in addition to the generated numerical value, and it is possible to confirm the temporal change of the quantitative value.

[0037] "Threshold information" is information indicating a threshold used when performing an evaluation regarding an evaluation item based on a subject image. The threshold information may be a fixed value set in advance, or a value updated at a predetermined frequency. Also, the threshold information is not limited to a single value, and it is also possible to set a plurality of thresholds according to the evaluation item, such as a first threshold and a second threshold. For example, when the first threshold is exceeded compared to the quantitative value, it is possible to output a warning as a state where there may be an abnormality, and when the second threshold is exceeded, it is possible to output a warning as an occurrence of an abnormality.

[0038] "Evaluation information" is information indicating the result of evaluation regarding evaluation items in a plurality of subject images taken multiple times over an arbitrary period. That is, as an example, the evaluation information is information indicating whether a quantitative value exceeding a threshold is detected in a quantitative value that changes over time, or information indicating the result of determining that an abnormality has occurred due to the detection of a quantitative value exceeding the threshold. The process of setting the threshold and the evaluation process will be described later.

[0039] 4. Processing Flow Executed by Server Device 100 FIG. 4 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, FIG. 4 is a diagram showing a series of processing flows for performing an evaluation regarding an evaluation item based on a subject image. The processing flow is mainly performed by the processor 111 of the server device 100 reading and executing a program stored in the memory 112. In the following, for convenience of explanation, in a work site in the secondary industry, based on a subject image obtained by photographing an article such as a tool and the work site where it is stored as a subject, the case of evaluating an evaluation item indicating the appropriateness of the management status (more specifically, the degree of disorder of the articles) will be described. However, the invention of the present application is not limited to this example only.

[0040] (A) Evaluation item input process According to FIG. 4, the processor 111 executes an evaluation item input process (S111). Specifically, the processor 111 acquires an evaluation item to be evaluated based on a subject image from the terminal device 300 via the communication interface 113. Then, the processor 111 newly generates evaluation item ID information, and stores the acquired evaluation item in association with the evaluation item ID information as input phrase information. Further, the processor 111 generates opposite phrase information and each feature information based on the input phrase information, and stores them in association with the evaluation item ID information.

[0041] Here, FIG. 5 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, FIG. 5 is a diagram showing a series of processing flows in the evaluation item input processing executed in S111 of FIG. 4. This processing flow is mainly performed by the processor 111 of the server device 100 reading and executing a program stored in the memory 112.

[0042] According to FIG. 5, the processor 111 acquires input phrase information, which is a phrase for specifying an evaluation item to be evaluated based on a subject image, from the terminal device 300 via the communication interface 113 (S211). As described with reference to FIG. 3, "appropriateness of the storage position of the article" is an example included in the input phrase information. When the processor 111 acquires the input phrase information that is an evaluation item, it newly generates evaluation item ID information and stores the input phrase information in the evaluation item management table in association with the generated evaluation item ID information.

[0043] Incidentally, such input phrase information, which is an evaluation item, is input as follows in the terminal device 300, for example. The processor 311 of the terminal device 300 receives a user's operation input via the input interface 313 and activates an application program that executes processing related to the evaluation. Then, when the application program is activated, the processor 311 receives a user's operation input via the input interface 313 and inputs an evaluation item (such as "appropriateness of the storage position of the article") for evaluating, for example, the appropriateness of the management status of an article such as a tool included as a subject in the subject image. The input may be performed by the input interface 313 receiving an operation input for the user to select an arbitrary character string (i.e., input of the character string "appropriateness of the storage position of the article"), or may be performed by receiving an operation input for selecting a desired one from a plurality of prepared options. Then, when the processor 311 receives the input of the evaluation item as described above, it transmits information indicating the evaluation item input via the communication interface 315 to the server device 100.

[0044] Next, the processor 111 of the server device 100 acquires opposite phrase information based on the input phrase information, which is the evaluation item acquired in S211 (S212). As an example of the opposite phrase information, a phrase such as "inappropriateness of the storage position of the article" can be cited. As an example of this process, there is a dictionary table in which opposite words are associated with each phrase in advance, and the processor 111 that has acquired the input phrase information returns to the dictionary table and performs a phrase conversion process. As another example of this process, the processor 111 generates a prompt (a question sentence such as "What is the phrase that has the opposite meaning to the phrase 'appropriateness of the storage position of the article'?"), and inputs the question sentence into a large language model stored in the server device 100 or another server device communicably connected thereto, and acquires an answer to the above prompt as an output from the large language model. When the processor 111 generates the opposite phrase information as described above, the processor 111 stores the input phrase information in the evaluation item management table in association with the evaluation item ID information.

[0045] Next, the processor 111 acquires feature information based on the input phrase information, which is the evaluation item acquired in S211 (S213). The feature information is visual features representing the phrase input as the input phrase information, that is, information indicating an index that is focused on when evaluating the degree of the evaluation item indicated by the phrase. Examples of the feature information include "the directions of the articles are aligned", "each article is arranged within the frame", "the articles are arranged in descending order of size", or "the total number of articles matches the number of frames serving as a guide for the arrangement". As an example of the process, there is a feature table in which a plurality of visual features are associated with each evaluation item in advance, and the process is performed by reading out the corresponding plurality of visual features by referring to the feature table based on the input phrase information acquired by the processor 111. As another example of the process, the processor 111 generates a prompt (a question sentence such as "What are the visual features representing the phrase 'appropriateness of the storage position of the article?'") based on the input phrase information, and inputs the question sentence into a large language model stored in the server device 100 or another server device communicably connected thereto, and acquires the answer to the above prompt from the large language model as an output. When the processor 111 generates feature information based on the input phrase information as described above, the processor 111 stores the feature information in the evaluation item management table in association with the evaluation item ID information.

[0046] Next, the processor 111 acquires feature information based on the opposite phrase information generated in S212 (S214). The feature information is visual features representing the phrase generated as the opposite phrase information, that is, information indicating an index that is focused on when evaluating the degree of the evaluation item indicated by the phrase. Examples of the feature information include "the orientations of the articles are not aligned", "each article is not arranged within the frame", "the sizes of the articles are in a random order", or "the total number of articles does not match the number of frames serving as a guide for the arrangement". An example of the process is to have a feature table in which a plurality of visual features are associated in advance for each evaluation item, and to read out the corresponding plurality of visual features by referring to the feature table based on the opposite phrase information acquired by the processor 111. Another example of the process is that the processor 111 generates a prompt (a question sentence such as "What are the visual features representing the phrase 'inappropriateness of the storage position of the articles'?") based on the opposite phrase information, and inputs the question sentence into a large language model stored in the server device 100 or another server device communicably connected thereto, and acquires an answer to the above prompt from the large language model as an output. When the processor 111 generates feature information based on the opposite phrase information as described above, the processor 111 stores the feature information in the evaluation item management table in association with the evaluation item ID information. Thus, the processing flow ends.

[0047] (B) Quantification process Returning to FIG. 4 again, when the evaluation item input process is performed as described above, the processor 111 executes a quantification process (S112). Specifically, the processor 111 acquires a subject image from an imaging device 200 that images articles such as tools as a subject at a work site in the secondary industry, and acquires a quantitative value by inputting the subject image into a learned model. The processor 111 stores the acquired subject image and the quantitative value obtained based thereon in the evaluation item management table in association with the evaluation item ID information.

[0048] Examples of the learned models include neural network-based methods such as neural networks, convolutional neural networks, multi-layer perceptrons (MLPs), long short-term memory (LSTM), gated recurrent units (GRUs), graph neural networks (GNNs), and transformers; methods using gradient boosting decision trees (GBDTs) such as LightGBM (Light Gradient Boosting Machine), XGBoost, and CatBoost; ridge regression, logistic regression, support vector regression (SVR), k-nearest neighbor method, decision trees, regression trees, random forests, etc. More specifically, it is possible to use learned models such as Vision Transformer and CLIP. For example, the processor 111 prepares a plurality of combinations of correct images (images with appropriate storage positions of articles) and incorrect images (images with inappropriate storage positions of articles) prepared in advance for the above-exemplified learning device, and causes the learning device to learn, thereby generating a learned model. Then, the processor 111 inputs a newly captured subject image to the learned model, and obtains a quantitative value as the accuracy similar to the correct image.

[0049] Here, FIG. 6 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, FIG. 6 is a diagram showing an example of the quantification process executed in S112 of FIG. 4. This processing flow is mainly performed by the processor 111 of the server device 100 reading and executing a program stored in the memory 112.

[0050] Note that in FIG. 6, the learned model for evaluating the evaluation items includes, as functional components, a text encoder for analyzing text information and outputting its feature amount as a vector, and an image encoder for analyzing an image and outputting its feature amount as a vector.

[0051] According to FIG. 6, the processor 111 first executes an input step. Specifically, the processor 111 inputs each piece of feature information (PT1 to PTn: for example, "the orientations of the articles are aligned", "each article is arranged within the frame", "the articles are arranged in order from the largest", or "the total number of articles matches the number of frames serving as a guide for the arrangement") based on the input phrase information stored in S213 of FIG. 5 to the text encoder of the learned model. Similarly, the processor 111 inputs each piece of feature information (NT1 to NTn: "the orientations of the articles are not aligned", "each article is not arranged within the frame", "the order of the sizes of the articles is chaotic", or "the total number of articles does not match the number of frames serving as a guide for the arrangement") based on the opposite phrase information stored in S214 of FIG. 5 to the text encoder of the learned model.

[0052] Also, in the work site in the secondary industry, from among the subject images taken with articles such as tools and the work site where they are stored as the subject, a plurality of correct images (PI1 to PIn) taken of the state where each article is appropriately stored and a plurality of incorrect images (NI1 to PIn) taken of the state where each article is inappropriately stored are stored in the memory 112 for learning. Then, the processor 111 reads out the correct images and the incorrect images (NI1 to PIn) from the memory 112 and inputs them to the image encoder.

[0053] Furthermore, the processor 111 receives the subject image (I) from the imaging device 200 via the communication interface 113 and stores it in the evaluation item management table in association with the evaluation item ID information for specifying the evaluation item to be evaluated based on the received subject image. Then, the processor 111 inputs the stored subject image (I) to the image encoder of the learned model. Note that the various parameters used for analysis in this image encoder are the same as those of the image encoder into which the correct image (PI1 to PIn) or the incorrect image (NI1 to NIn) is input.

[0054] Next, the processor 111 executes a vector output step. Specifically, when each piece of feature information (PT1 to PTn) based on the input phrase information is input to the text encoder in the input step, the processor 111 outputs, as vectors, the feature amounts that are the results of analyzing each of PT1 to PTn from the text encoder. Also, when each piece of feature information (NT1 to NTn) based on the opposite phrase information is input to the text encoder in the input step, the processor 111 outputs, as vectors, the feature amounts that are the results of analyzing each of NT1 to NTn from the text encoder.

[0055] Similarly, when the correct images for learning (PI1 to PIn) are input to the image encoder in the input step, the processor 111 outputs, as vectors, the feature amounts that are the results of analyzing each of PI1 to PIn from the image encoder. Also, when the incorrect images (NI1 to NIn) are input to the image encoder in the input step, the processor 111 outputs, as vectors, the feature amounts that are the results of analyzing each of NI1 to NIn from the image encoder.

[0056] Furthermore, when the subject image (I) to be quantified is input to the image encoder in the input step, the processor 111 outputs, as a vector, the feature amount that is the result of analyzing the subject image from the image encoder. [[ID=,10]]

[0057] Next, the processor 111 executes a similarity calculation step. Specifically, the processor 111 calculates the average value (average PT) of each feature amount (vector) corresponding to PT1 to PTn obtained in the vector output step. Also, the processor 111 calculates the average value (average NT) of each feature amount (vector) corresponding to NT1 to NTn obtained in the vector output step.

[0058] Similarly, the processor 111 calculates the average value (average PI) of each feature amount (vector) corresponding to PI1 to PIn obtained in the vector output step. Further, the processor 111 calculates the average value (average NI) of each feature amount (vector) corresponding to NI1 to ITn obtained in the vector output step.

[0059] Then, when each average value is calculated, the processor 111 calculates the similarity between the calculated average value and the feature amount (vector) calculated based on the subject image. Specifically, the processor 111 calculates the PT similarity, which is the similarity between the feature amount (vector) calculated based on the subject image and the average PT, which is the average value of each feature amount (vector) corresponding to PT1 to PTn. Further, the processor 111 calculates the NT similarity, which is the similarity between the feature amount (vector) calculated based on the subject image and the average NT, which is the average value of each feature amount (vector) corresponding to NT1 to NTn. Further, the processor 111 calculates the PI similarity, which is the similarity between the feature amount (vector) calculated based on the subject image and the average PI, which is the average value of each feature amount (vector) corresponding to PI1 to PIn. Furthermore, the processor 111 calculates the NI similarity, which is the similarity between the feature amount (vector) calculated based on the subject image and the average NI, which is the average value of each feature amount (vector) corresponding to NI1 to NIn.

[0060] That is, the PT similarity indicates the degree to which the visual features included in the subject image are similar to the visual features (feature information) generated from the input phrase information, and the NT similarity indicates the degree to which the visual features included in the subject image are similar to the visual features (feature information) generated from the opposite phrase information. That is, the higher the PT similarity, the more appropriate the storage position of the item included in the subject image, and the higher the NT similarity, the more inappropriate the storage position of the item included in the subject image. Similarly, the PI similarity indicates the degree to which the visual features included in the subject image are similar to the visual features (feature information) included in the correct image for learning, and the NI similarity indicates the degree to which the visual features included in the subject image are similar to the visual features (feature information) included in the incorrect image for learning. That is, the higher the PI similarity, the more appropriate the storage position of the item included in the subject image, and the higher the NI similarity, the more inappropriate the storage position of the item included in the subject image.

[0061] Next, the processor 111 executes a quantitative value output step. Specifically, the processor 111 performs processing to convert the probability that the subject image is similar to the visual features generated from the input phrase information and the probability that the subject image is similar to the visual features generated from the opposite phrase information based on each numerical value calculated in the similarity calculation step. Similarly, the processor 111 performs processing to convert the probability that the subject image is similar to the correct image and the probability that the subject image is similar to the incorrect image based on each numerical value calculated in the similarity calculation step.

[0062] An example of such conversion processing is the softmax function. By giving the PT similarity and the NT similarity to the softmax function, the softmax function can output a quantitative value (a value indicating probability) that indicates which of the visual features generated from the input phrase information and the visual features generated from the opposite phrase information the subject image is closer to. Similarly, by giving the PI similarity and the NI similarity to the softmax function, the softmax function can output a quantitative value (a value indicating probability) that indicates which of the correct image and the incorrect image the subject image is closer to.

[0063] The processor 111 processes each of the obtained quantitative values and generates a final output value. The processor 111 stores the generated output value as quantitative value information in association with the evaluation item ID information. Note that, as an example, the final output value is obtained by weighted-averaging each quantitative value. However, naturally, this method is just an example, and other methods such as summation or multiplication may be used. Thus, the said processing flow ends.

[0064] (C) Time-series evaluation processing Returning again to FIG. 4, when the quantification processing is performed as described above, the processor 111 executes time-series evaluation processing (S113). Specifically, the processor 111 refers to the evaluation item management table, reads out the quantitative value information stored in time series from the evaluation item ID information in which the subject image subjected to the quantification processing is stored in association, and performs an evaluation regarding the said evaluation item. Then, the processor 111 stores the evaluated result as evaluation information in association with the evaluation item ID information in the evaluation item management table.

[0065] (C-1) Threshold setting processing used for evaluation The threshold is used when performing an evaluation regarding the evaluation item as described above, and although the said threshold can also be set to a fixed value determined in advance. However, in the present embodiment, it is desirable to ignore single occurrences of anomalies (for example, when noise accidentally enters the subject or a person suddenly enters), and to detect anomalies that are continuous in time series. Therefore, it is desirable to flexibly set the threshold using the quantitative values acquired in time series. FIG. 7 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, FIG. 7 is a diagram showing the flow of processing for setting the "threshold" used in the time-series evaluation processing executed in S113 of FIG. 4. The said processing flow is performed, for example, at a frequency of once a day, mainly by the processor 111 of the server device 100 reading out and executing a program stored in the memory 112.

[0066] According to FIG. 7, the processor 111 refers to the quantitative value information in the evaluation item management table at a predetermined cycle (for example, once a day), and sets a threshold value used for evaluation for each evaluation item ID information. Specifically, the processor 111 reads out the quantitative value information for a predetermined period from each piece of quantitative value information associated with each evaluation item ID information (S311). This period may be a predetermined period such as the past 10 days, or may be the entire period stored as quantitative value information.

[0067] Next, the processor 111 refers to the evaluation information corresponding to the read quantitative value information, and checks whether an evaluation result indicating "abnormality" is stored within a predetermined period (S312). If the processor 111 determines that an evaluation result indicating "abnormality" is not stored within the predetermined period, that is, if the result in S312 is "No", the processor 111 ends the process without updating the threshold value information stored for each evaluation item in the evaluation item management table.

[0068] On the other hand, if the processor 111 determines that an evaluation result indicating "abnormality" is stored within the predetermined period, it determines whether the logarithms of the respective quantitative values for the predetermined period read in S311 follow a normal distribution (S313). Examples of such determination processes include the Kolmogorov-Smirnov test method, the Shapiro-Wilk test method, and the Anderson-Darling test method, and any of these methods may be adopted. If the processor 111 determines that the logarithms of the respective quantitative values follow a normal distribution as a result of the above determination (S314), it calculates a Z-score from the logarithms of the respective quantitative values (S319).

[0069] On the other hand, when the processor 111 determines that the logarithms of the respective quantitative values do not follow a normal distribution as a result of the above determination (S314), it determines whether each quantitative value follows a normal distribution (S315). As an example of such a determination process, similar to the above, the Kolmogorov-Smirnov test method, the Shapiro-Wilk test method, the Anderson-Darling test method, etc. can be mentioned, and any of these methods may be adopted. When the processor 111 determines that each quantitative value follows a normal distribution as a result of the above determination (S316), it calculates a Z-score from each quantitative value (S318).

[0070] On the other hand, when the processor 111 determines that each quantitative value does not follow a normal distribution as a result of the above determination (S316), it calculates a robust Z-score for each quantitative value (S317).

[0071] Note that the Z-score is a value obtained by converting the distribution of quantitative values for a predetermined period so that the average value is 0 and the standard deviation is 1. Also, the robust Z-score is a value used for the normalization or standardization of data showing a non-normal distribution, and is obtained by replacing the average value of the Z-score with the median of the distribution of quantitative values for a predetermined period and the standard deviation with the interquartile range.

[0072] When each value is calculated, the processor 111 sets a threshold value based on each calculated value. Specifically, when the processor 111 calculates a robust Z-score in S317, it sets, as a threshold value, a quantitative value such that the robust Z-score becomes a predetermined value (for example, "3") determined in advance, and stores this value in the threshold information of the evaluation item management table in association with the evaluation item ID information (S320). Also, when the processor 111 calculates a Z-score in S318 or S319, it sets, as a threshold value, a quantitative value such that the Z-score becomes a predetermined value (for example, "3") determined in advance, and stores this value in the threshold information of the evaluation item management table in association with the evaluation item ID information (S321). Thus, the processing flow ends.

[0073] (C-2) Evaluation process regarding evaluation items FIG. 8 is a diagram showing a processing flow executed in the server device 100 according to an embodiment of the present disclosure. Specifically, FIG. 8 is a diagram showing the flow of a determination process using a threshold value in the time-series evaluation process executed in S113 of FIG. 4. This processing flow is mainly performed by the processor 111 of the server device 100 reading and executing a program stored in the memory 112 each time a subject image is newly acquired and the processing of FIG. 6 is performed.

[0074] According to FIG. 8, the processor 111 refers to the quantitative value information in the evaluation item management table and reads out, in time series, the quantitative value information for a predetermined period including the latest quantitative value associated with the evaluation item ID information of the evaluation item to be evaluated. This period may be a predetermined period such as the past 10 days, or may be the entire period stored as quantitative value information. When the processor 111 reads out the quantitative value information, it calculates a moving average from the read time-series quantitative value information (S311). As a method for calculating the moving average, there are methods such as a simple moving average, a weighted moving average, and an exponential moving average, and any of them may be used. The moving average is a preferable value for smoothing data arranged in time series like the quantitative value information.

[0075] Next, the processor 111 refers to the threshold value information associated with the evaluation item ID information and determines whether the calculated moving average is higher than the threshold value (S412). Then, when the processor 111 determines that it is higher, it generates an evaluation result (for example, the management status of the article is inappropriate) indicating that an abnormality has been continuously detected in each subject image taken during the predetermined period, and stores it in association with the evaluation item ID information as evaluation information (S413). On the other hand, when the calculated moving average is equal to or lower than the threshold value with reference to the threshold value information associated with the evaluation item ID information, the processor 111 skips the process according to S413. Thus, the processing flow ends.

[0076] In FIG. 8, only a single threshold value is described, but it is also possible to use a plurality of threshold value information. For example, when the moving average exceeds the first threshold value, a warning may be output as a state where there is a risk of abnormality, and when it exceeds the second threshold value, a warning may be output as an indication that an abnormality has occurred.

[0077] Also, in FIG. 8, it is described that an abnormality is evaluated when it is higher than the threshold value, but it may be determined that an abnormality has occurred when it is lower than the threshold value.

[0078] Also, in FIG. 8, a moving average is used for processing time-series quantitative value information, but of course, the quantitative value information may be processed by other methods. For example, the processor 111 may evaluate that an abnormality has occurred when the threshold value is exceeded continuously a predetermined number of times based on the quantitative value information within a predetermined period.

[0079] (D) Output processing of evaluation results Returning to FIG. 4 again, when the time-series evaluation process is performed as described above, the processor 111 executes output processing of the evaluation result (S114). Specifically, the processor 111 outputs evaluation information associated with the evaluation item ID information of the evaluation item to the terminal device 300 that transmitted the evaluation item in S111 via the communication interface 113. Thus, a series of processing flows for performing an evaluation regarding the evaluation item based on the subject image is completed.

[0080] Although not particularly shown, when the terminal device 300 receives the evaluation information from the server device 100 via the communication interface 315, the terminal device 300 outputs the evaluation information received via the output interface 314.

[0081] As described above, in the present embodiment, it is possible to provide a processing device, a processing program, and a processing method capable of more reliable evaluation. Further, in the present embodiment, in the evaluation regarding the evaluation items based on the subject image, since a quantitative value is used, a uniform evaluation using a threshold value or the like can be performed. Further, in the present embodiment, since the threshold value used for evaluation is calculated from the quantitative values for a predetermined period, the influence of momentary abnormalities such as noise can be reduced. Further, in the present embodiment, in the evaluation regarding the evaluation items, since time-series quantitative values are used, it is possible to evaluate the presence or absence of a true abnormality while ignoring momentary abnormalities such as noise to some extent.

[0082] In the above, the case of evaluating the presence or absence of "abnormality" as evaluation information has been described. However, instead of this, it may be possible to evaluate whether it is in a "normal" state, or it is also possible to evaluate various states such as "dirtiness", "peak (such as of a flower)", and "ripeness of the harvested product".

[0083] Further, in the above, the "appropriateness of the storage position of the article" has been described as an example of the evaluation item, but the same processing can be performed for other evaluation items such as "dirtiness of the room", "dirtiness on the desk", "ripeness of the harvested product", and "degree of blooming of cherry blossoms".

[0084] The processes and procedures described in this specification can be realized not only by those explicitly described in the embodiment, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described in this specification are realized by implementing the logic corresponding to the process in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, a magnetic disk, or an optical storage. Further, the processes and procedures described in this specification can be implemented as a computer program for those processes and procedures and executed on various computers including a processing device and a server device.

[0085] Even if it is described that the processes and procedures described in this specification are executed by a single device, software, component, or module, such processes or procedures can be executed by a plurality of devices, a plurality of software, a plurality of components, and / or a plurality of modules. Also, even if it is described that various information described in this specification is stored in a single memory or storage unit, such information can be distributed and stored in a plurality of memories provided in a single device or a plurality of memories distributed among a plurality of devices. Furthermore, the software and hardware elements described in this specification can be realized by integrating them into fewer components or decomposing them into more components.

Description of Reference Numerals

[0086] 1 Processing system 100 Server device 200 Imaging device 300 Terminal device

Claims

1. A processing device including at least one processor, wherein the at least one processor, obtains a subject image via an imaging device configured to image one or more objects as subjects via a communication interface, obtains an evaluation item to be evaluated in the obtained subject image based on a user's operation input, performs an evaluation regarding the evaluation item based on the obtained subject image by using a learned model configured to perform an evaluation regarding the evaluation item based on the obtained subject image, and outputs a result of the evaluation regarding the evaluation item, and is configured to execute processing therefor.

2. The processing device according to claim 1, wherein the result is a value quantified regarding the evaluation item.

3. The processing device according to claim 2, wherein the at least one processor is configured to execute processing for evaluating an abnormality based on a comparison between the quantified value and a predetermined threshold value.

4. The at least one processor, obtains a moving average of the quantified value at present based on time series information obtained by obtaining the quantified value at a predetermined interval in an arbitrary period up to the present, and evaluates an abnormality based on a comparison between the obtained moving average and a predetermined threshold value, and is configured to execute processing therefor, and is the processing device according to claim 2.

5. The processing device according to claim 4, wherein the threshold value is converted into different values based on whether or not a plurality of quantified values obtained in an arbitrary period follow a normal distribution.

6. The processing device according to claim 1, wherein the at least one processor is configured to execute processing for generating feature information indicating visual features for expressing the evaluation item based on the obtained evaluation item.

7. The processing device according to claim 1, wherein the at least one processor is configured to execute processing for generating visual features for expressing each of a first phrase which is a character string indicating the evaluation item and a second phrase which is a character string having a meaning opposite to that of the first phrase.

8. The processing device according to claim 1, wherein the subject image is evaluated based on the similarity between each feature amount obtained from the learning correct image and the learning incorrect image input to the learned model and the feature amount obtained from the subject image input to the learned model.

9. The processing device according to claim 1, wherein the subject is a work site in the primary industry or the secondary industry.

10. The processing device according to claim 1, wherein the evaluation item is an item related to the appropriateness of the management status in the one or more objects.

11. The one or more objects are articles used at the work site, and the evaluation item is an item related to the degree of disorder of the article. The processing device according to claim 9.

12. By being executed by at least one processor, acquire a subject image via a photographing device configured to photograph one or more objects as a subject via a communication interface, acquire an evaluation item to be evaluated in the acquired subject image based on a user's operation input, evaluate the evaluation item based on the acquired subject image using a learned model configured to evaluate the evaluation item based on the acquired subject image, output the result of the evaluation regarding the evaluation item, A processing program for causing the at least one processor to function as described above.

13. A processing method executed by at least one processor, comprising: acquiring a subject image via a photographing device configured to photograph one or more objects as a subject via a communication interface; acquiring an evaluation item to be evaluated in the acquired subject image based on a user's operation input; evaluating the evaluation item based on the acquired subject image using a learned model configured to evaluate the evaluation item based on the acquired subject image; outputting the result of the evaluation regarding the evaluation item; A processing method including the above steps.

Images