Processing device, processing program and processing method

Abstract

To provide a processing device, a processing program, and a processing method suitable for analyzing property information of a fluid.SOLUTION: A processing device is provided that is configured to perform processing to: acquire image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; input the acquired image data into a trained property estimation model for generating property information indicating at least one of speed and direction of a fluid, thereby estimating the property information for one or more pixels constituting the image data; calculate an index indicating reliability of the estimated property information for the one or more pixels; and correct the property information of the one or more pixels based on property information generated for one or more pixels different from the one or more pixels, based on the calculated reliability.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a processing device, a processing program, and a processing method for analyzing characteristic information of a fluid from an image captured by a camera. [Background technology]

[0002] It has been known in the past to input an image captured by a camera or the like into a trained model to identify an object contained in the image. For example, Patent Document 1 describes an object state identification device for detecting an object region that includes an object and has a predetermined shape in an image by inputting an image captured by a camera into a first classifier that has been trained in advance to detect a predetermined object. However, such a device is limited to identifying objects with little change in shape. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-128705 Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above-mentioned technology, an object of the present disclosure is to provide, in various embodiments, a processing device, a processing program, and a processing method suitable for analyzing characteristic information of a fluid. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a processing device having at least one processor, wherein the at least one processor is configured to perform processing for acquiring image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface, and inputting the acquired image data into a learned characteristic estimation model for generating characteristic information indicating at least one of the speed and direction of a fluid, thereby estimating the characteristic information for one or more pixels constituting the image data, calculating an index indicating the reliability of the estimated characteristic information for the one or more pixels, and correcting the characteristic information of the one or more pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability.

[0006] According to one aspect of the present disclosure, there is provided a processing program that, when executed by at least one processor provided in a processing device, causes the at least one processor to function as follows: acquire image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface, input the acquired image data into a learned characteristic estimation model for generating characteristic information indicating at least one of the speed and direction of a fluid, estimate the characteristic information for one or more pixels constituting the image data, calculate an index indicating the reliability of the estimated characteristic information for each of the one or more pixels, and correct the characteristic information of the one or more pixels based on characteristic information generated in one or more other pixels different from the one or more pixels based on the calculated reliability.

[0007] According to one aspect of the present disclosure, there is provided a processing method executed by at least one processor provided in a processing device, the processing method including: acquiring image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; estimating the characteristic information for each of one or more pixels constituting the image data by inputting the acquired image data into a learned characteristic estimation model for generating characteristic information indicating at least one of the speed and direction of a fluid; calculating an index indicating the reliability of the estimated characteristic information for each of the one or more pixels; and correcting the characteristic information of the one or more pixels based on characteristic information generated in one or more other pixels different from the one or more pixels based on the calculated reliability. Effect of the Invention

[0008] According to the present disclosure, it is possible to provide a processing device, a processing program, and a processing method suitable for analyzing characteristic information of a fluid.

[0009] It should be noted that the above effects are merely illustrative for the convenience of explanation and are not limiting. In addition to or in place of the above effects, any effect described in this disclosure or any effect obvious to a person skilled in the art may be achieved. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of use of a processing system 1 according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of a processing system 1 according to one embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram showing a configuration of the processing device 100 according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a diagram conceptually illustrating an image table stored in the memory 113 of the processing device 100 according to an embodiment of the present disclosure. [Diagram 5]FIG. 5 is a diagram showing a process flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating a process flow for generating a trained model according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating a process flow for generating a trained model according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating a process flow for generating a trained model according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing a process flow executed in the processing device 100 according to an embodiment of the present disclosure. [Figure 10A] FIG. 10B is a diagram showing an example of a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. [Figure 10B] FIG. 10B is a diagram showing an example of characteristic information generated from a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. [Figure 10C] FIG. 10C is a diagram showing an example in which characteristic information generated from a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure is corrected. [Figure 11] FIG. 11 is a diagram showing an example of a screen output from the processing device 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which common elements are designated by the same reference numerals.

[0012] 1. Overview of Processing System 1 The processing system 1 according to the present disclosure is used to identify an area in an image captured by an imaging device such as a camera where a specific subject exists, and to analyze characteristic information of the subject existing in the area. The subject used in the processing system 1 includes a fluid that can be easily deformed by the application of external pressure. Examples of such fluids include gas, liquid, a mixture of gas and liquid, and a mixture of gas or liquid with a small amount of solid. Such fluids generally have viscosity, move due to an external force, and have an inconsistent boundary with other objects.

[0013] FIG. 1 is a diagram showing an example of use of a processing system 1 according to an embodiment of the present disclosure. Specifically, FIG. 1 is a diagram conceptually showing a captured image in which a fluid used in the processing system 1 is captured as a subject. According to FIG. 1, a pipe 11 having a hollow interior and configured to allow a fluid to pass through, a protrusion 12 formed to protrude from the inner circumferential surface side of the pipe 11 to the inside, and a fluid 14 passing through the inside of the pipe 11 are used as subjects. Then, by applying a captured image of these subjects captured by a camera or the like to the processing system 1 according to the present disclosure, a region in which the fluid 14 exists in the captured image is identified, and characteristic information of the fluid 14 existing in the region is analyzed.

[0014] Here, in the processing system 1 according to an embodiment of the present disclosure, it is also possible to further correct the analyzed characteristic information of the fluid 14 to obtain more accurate characteristic information of the fluid 14. For example, in FIG. 1, the fluid 14 flows at a constant speed from left to right (the direction indicated by the arrow in FIG. 1) inside the pipe 11. At this time, the fluid 14 passes over the protrusion 12 in the pipe 11. Therefore, for example, in the region 13a in front of the protrusion 12, the flow speed of the fluid 14 temporarily increases, and the flow direction also changes along the shape of the protrusion 12. In addition, even after passing over the protrusion 12, the fluid behaves like a vortex in the region 13b near the rising edge of the protrusion 12. Therefore, in the processing system 1, it is possible to obtain more accurate characteristic information by correcting the characteristic information in consideration of such changes in the behavior of the fluid.

[0015] In the present disclosure, the term "fluid" refers to, as described above, gas, liquid, a mixture of gas and liquid, a mixture of gas or liquid with a small amount of solid, etc. Examples of such fluids include water, beverages, jelly, cosmetics, medicines, food, cement, foam, edible oil, fuel oil, industrial oil, etc.

[0016] In addition, examples of characteristic information in the present disclosure include the viscosity, flow velocity, direction, velocity vector, mass, pressure, temperature, or a combination thereof, preferably the flow velocity, direction, velocity vector, or a combination thereof, more preferably the velocity vector, of the fluid. The processing system 1 does not need to analyze all of these pieces of characteristic information, but it is sufficient if it can analyze at least one of them.

[0017] In addition, in the present disclosure, the captured image may be one or more videos or one or more still images. In order to obtain such captured images, an imaging device such as a camera is used. As an example of the operation, when the power button is pressed, a through image is captured by the imaging device, and the captured through image is displayed on the display of the imaging device. After that, when the imaging button is pressed by the user, one or more still images are captured by the imaging device, and the captured image is displayed on the display. Or, when the imaging button is pressed by the user, capturing of a video is started, and the image captured by the imaging device during that time is displayed on the display. Then, when the imaging button is pressed again, capturing of the video is terminated. In this way, in a series of operations, various images such as through images, still images, and videos are captured by the imaging device and displayed on the display, but the captured image does not mean only a specific image among them, but may include all images captured by the imaging device. Furthermore, in the present disclosure, image data merely means data that shows a captured image in a predetermined data format, and does not refer only to data shown in a specific data format.

[0018] 2. Configuration of Processing System 1 Fig. 2 is a schematic diagram of a processing system 1 according to an embodiment of the present disclosure. According to Fig. 2, the processing system 1 includes a processing device 100 and an imaging device 200 communicably connected to the processing device 100 via a wired or wireless network. The processing device 100 receives an operation input by a user and controls imaging by the imaging device 200. The processing device 100 also processes the captured image captured by the imaging device 200 to identify an area in which a fluid exists in the captured image and analyze property information of the fluid.

[0019] A typical example of such an imaging device 200 is a camera, but it is not necessarily required to be connected to the processing device 100 via a wired or wireless network as shown in FIG. 2. For example, even an imaging device 200 provided as a component of the processing device 100 can appropriately execute the processing according to the present disclosure. In addition, the imaging device 200 does not need to exist alone, and may exist in combination with other devices. For example, the imaging device 200 can be disposed in a manufacturing line device for manufacturing a fluid, a product containing a fluid, or a product made of a fluid.

[0020] FIG. 3 is a block diagram showing a configuration of a processing device 100 according to an embodiment of the present disclosure. According to FIG. 3, the processing device 100 includes an output interface 111, a processor 112, a memory 113, an input interface 115, and a communication interface 114. These components are electrically connected to each other via control lines and data lines. Note that the processing device 100 does not need to include all of the components shown in FIG. 3, and may be configured by omitting some of the components, or by adding other components. For example, the processing system 1 may include a battery for driving each component.

[0021] Moreover, as such a processing device 100, a laptop PC, a desktop PC, a smartphone, a tablet, or the like can be used. However, without being limited to these, other devices such as a server device can also be suitably used as the processing device 100. Moreover, the processing device 100 does not need to execute all the processes by itself, and can execute some of the processes in cooperation with other devices. For example, analysis processes using a learned algorithm can be executed jointly in a cloud server device, and in such a case, the processing device 100 may include the device.

[0022] First, the processor 112 functions as a control unit that controls other components of the processing device 100 based on a program stored in the memory 113. The processor 112 acquires image data of a captured image from the imaging device 200 based on the program stored in the memory 113, and controls processing related to analysis of characteristic information of a fluid. Specifically, the processor 112 executes, based on the program stored in the memory 113, "processing of acquiring image data of a captured image captured by the imaging device 200 included in the processing device 100 or connected via the communication interface 114", "processing of inputting image data of the captured image to the learned fluid region identification model", "processing of generating fluid region identification information in which a fluid region in which a fluid exists in the captured image is identified based on the acquired image data", "processing of analyzing classification information of a fluid present in a fluid region based on the fluid region identification information", "processing of estimating characteristic information of a fluid present in a fluid region based on the analyzed classification information", "processing of calculating an index indicating the reliability of characteristic information estimated for one or more pixels", "processing of correcting characteristic information of a plurality of pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability", and the like. The processor 112 may be mainly composed of one or more processors, and such processors may be a combination of a CPU, a GPU, an FPGA, and the like.

[0023] The memory 113 is composed of a RAM, a ROM, a non-volatile memory, an SSD, an HDD, etc., and functions as a storage unit. The memory 113 stores instructions and commands for various controls of the processing device 100 according to this embodiment as a program. Specifically, the memory 113 stores programs for the processor 112 to execute, such as "processing of acquiring image data of a captured image captured by the imaging device 200 included in the processing device 100 or connected via the communication interface 114", "processing of inputting image data of the captured image to the learned fluid region identification model", "processing of generating fluid region identification information in which a fluid region in the captured image in which a fluid exists is identified based on the acquired image data", "processing of analyzing classification information of a fluid existing in a fluid region based on the fluid region identification information", "processing of estimating characteristic information of a fluid existing in a fluid region based on the analyzed classification information", "processing of calculating an index indicating the reliability of characteristic information estimated for one or more pixels", and "processing of correcting characteristic information of a plurality of pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability". In addition to the program, the memory 113 also stores image data of captured images captured by the imaging device 200, an image table for managing the images, etc. The memory 113 also stores programs related to each trained model, such as a trained segmentation judgment model, a trained fluid region identification model, a trained fluid classification model, and a trained characteristic estimation model, which are used to analyze the captured images.

[0024] The input interface 115 functions as an input unit that accepts user's instruction input to the processing device 100. Examples of the input interface 115 include an "imaging button" that transmits an instruction to image a subject in the imaging device 200, a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, a cross key button for moving a pointer or the like displayed on the display via the output interface 111, an on / off key for turning on and off the power of the processing device 100, and physical key buttons such as character input key buttons for inputting various characters. Note that the input interface 115 may be a touch panel that is superimposed on the display functioning as the output interface 111 and has an input coordinate system corresponding to the display coordinate system of the display. In this case, icons corresponding to the physical keys are displayed on the display, and a selection is made for each icon by the user inputting an instruction via the touch panel. The method of detecting the user's instruction input by the touch panel may be any method, such as a capacitance type or a resistive film type. The input interface 115 does not always need to be physically provided in the processing device 100, and may be connected as necessary via a wired or wireless network.

[0025] The output interface 111 functions as an output unit for outputting an image captured by the imaging device 200, a segmentation image processed by the processing device 100, an analyzed fluid image, and the like. An example of the output interface 111 is a display configured with a liquid crystal panel, an organic EL display, a plasma display, or the like. However, it is not necessarily required that the processing device 100 itself is provided with a display. For example, an interface for connecting to a display or the like connectable to the processing device 100 via a wired or wireless network can also function as the output interface 111 for outputting display data to the display or the like.

[0026] The communication interface 114 functions as a communication unit for transmitting and receiving various commands related to the start of imaging and the like, and image data of an image captured by the imaging device 200, to the imaging device 200 connected via a wired or wireless network. Examples of the communication interface 114 include a wired communication connector such as USB or SCSI, a wireless communication transmission / reception device such as LTE, wireless LAN, Bluetooth (registered trademark), or infrared, and various connection terminals for a printed circuit board or a flexible circuit board.

[0027] Although not specifically shown, the imaging device 200 includes a camera, a light source, a processor, a memory, a display panel, an input interface, a communication interface, etc. These components are electrically connected to each other via control lines and data lines. Note that the imaging device 200 does not need to include all of these components, and it is possible to omit some of them or to add other components.

[0028] In the imaging device 200, the camera functions as an imaging unit that detects light reflected from a subject and generates a subject image. The camera includes, for example, a CMOS image sensor to detect the light, and a lens system and a drive system to realize a desired function. The image sensor is not limited to a CMOS image sensor, and other sensors such as a CCD image sensor can also be used.

[0029] The light source is driven by an instruction from the processing device 100 or the imaging device 200, and functions as a light source unit for irradiating light into the oral cavity. The light source includes one or more light sources. In this embodiment, the light source is composed of one or more LEDs, and light having a predetermined frequency band is irradiated from each LED in the direction of the oral cavity. As an example, the light source uses light having a desired band from the ultraviolet light band, the visible light band, and the infrared light band, or a combination thereof.

[0030] The processor functions as a control unit that controls other components of the imaging device 200 based on a program stored in the memory. Specifically, the processor controls the driving of the camera and the driving of the light source based on the program stored in the memory, and controls the storage in the memory of the captured image captured by the camera and the imaging conditions for capturing the image. The processor also controls the output of the captured image stored in the memory to the display panel and the transmission to the processing device 100. The processor is mainly composed of one or more processors, and a CPU is used as an example of such a processor.

[0031] The memory is composed of RAM, ROM, non-volatile memory, SSD, HDD, etc., and functions as a storage unit. The memory stores instructions and commands for various controls of the imaging device 200 as programs. In addition to the programs, the memory also stores captured images captured by the camera, etc.

[0032] The display panel functions as a display unit for displaying a subject image captured by the imaging device 200. The display panel is configured by a liquid crystal panel, but is not limited to a liquid crystal panel and may be configured by an organic EL display, a plasma display, or the like.

[0033] The input interface functions as an input unit that accepts user's instruction input to the processing device 100 and the imaging device 200. Examples of the input interface include a "capture button" for instructing the imaging device 200 to start and end recording, a "power button" for turning the imaging device 200 on and off, a "confirmation button" for making various selections, a "back / cancel button" for returning to the previous screen or canceling an input confirmation operation, and physical key buttons such as a cross key button for moving icons displayed on the display panel. Note that these various buttons and keys may be physically prepared, or may be displayed as icons on the display panel and selectable using a touch panel or the like superimposed on the display panel as an input interface. The method of detecting the user's instruction input using the touch panel may be any method such as a capacitance type or a resistive film type.

[0034] The communication interface functions as a communication unit for transmitting and receiving information to and from the imaging device 200 and / or other devices. Examples of the communication interface include a connector for wired communication such as USB or SCSI, a transmitting and receiving device for wireless communication such as wireless LAN, Bluetooth (registered trademark), or infrared, and various connection terminals for printed circuit boards or flexible circuit boards.

[0035] Furthermore, the processing system 1 may include various sensor devices, although not shown. Such sensor devices may be any devices capable of measuring the pressure applied to the subject, the gravity applied to the subject, the surface condition of the subject, the temperature of the surroundings in which the subject is placed, and the like. Specifically, an image sensor for analyzing the surface condition of the subject, a temperature sensor for measuring the temperature around the subject, a temperature sensor for measuring the temperature inside the subject, a pressure sensor, a distance sensor, or a combination of these may be included. The sensor device may exist independently and transmit a measured output value to the processing device 100. Alternatively, the sensor device may be provided inside the processing device 100 or the imaging device 200 as one of the components thereof, and transmit a measured output value so that it can be processed by the processor 112 of the processing device 100.

[0036] 3. Information stored in memory 113 of processing device 100 4 is a diagram conceptually illustrating an image table stored in memory 113 of processing device 100 according to an embodiment of the present disclosure. Information stored in the image table is updated and stored as needed in accordance with the progress of processing by processor 112 of processing device 100.

[0037] 4, the image table stores image data, imaging condition information, imaging environment information, segmentation image data, classification information, estimated characteristic information, corrected characteristic information, etc., in association with image ID information. "Image ID information" is information for identifying each captured image captured by the imaging device 200. The image ID information is information that can be generated each time a new captured image is captured by the imaging device 200 or each time a new captured image is acquired by the processing device 100.

[0038] "Image data" is information on the image data of an image captured by the imaging device 200 or information that specifies the storage destination. Here, the captured image includes one or more pixels, and the image data is composed of pixel values ​​calculated for each pixel or a predetermined number of pixels among the one or more pixels. Note that such image data also includes image data in which image processing such as sharpening has been performed on the captured image data. Furthermore, in this disclosure, the terms "per pixel" and "each pixel" are used, and these may mean only a single pixel or may mean "a pixel group consisting of a predetermined number of pixels."

[0039] The "imaging condition information" is information indicating the imaging conditions when the captured image is captured by the imaging device 200. Examples of such imaging condition information include the aperture value, shutter speed, ISO sensitivity, exposure, white balance, or a combination thereof when the captured image is captured. The imaging condition information is stored in the imaging device 200 as the imaging conditions, and is acquired when image data of the captured image is received from the imaging device 200. In the imaging device 200, such imaging conditions can be set automatically, or can be set according to an instruction input from the user or another user accepted by the processing device 100 or the imaging device 200. The "imaging environment information" is information about the subject itself or the environment around the subject when the captured image is captured by the imaging device 200. Examples of such imaging environment information include output values ​​measured by various sensor devices, such as pressure (atmospheric pressure), gravity applied to the subject, surface condition of the subject, temperature of the surroundings where the subject is placed, and distance to the subject. The imaging environment information is measured by various sensors in synchronization with the imaging instruction being accepted by the imaging device 200, and measurement instructions are accepted by the various sensors. The measured output value is then acquired by the processing device 100 via the communication interface 114 .

[0040] The "segmentation image data" is image data obtained as a result of inputting the image data of the captured image into the trained segmentation judgment model. As an example, it is image data in which color information is added to the result of segmenting (dividing) a fluid region where a fluid exists, other regions other than the fluid region, and a boundary region that is the boundary between them, making each region identifiable. The "classification information" is one of the attribute information of the fluid, and is information indicating the type of fluid obtained as a result of inputting the image data of the captured image and the segmentation image data into the trained fluid classification model. Examples of such fluid types include water, beverages, jelly, cosmetics, medicines, food, cement, foam, edible oil, fuel oil, industrial oil, etc. The "estimated characteristic information" is one of the attribute information of the fluid, and is information indicating the characteristics of the fluid obtained as a result of inputting the image data of the captured image, the segmentation image data, and the classification information into the trained characteristic estimation model. Examples of such fluid characteristic information include the viscosity, flow velocity, direction, velocity vector, mass, pressure, temperature, or a combination thereof, preferably the flow velocity, direction, velocity vector, or a combination thereof, more preferably the velocity vector, etc. The "corrected characteristic information" is,

[0041] 4. Processing flow executed by the processing device 100 (A) Processing related to the acquisition of characteristic information Fig. 5 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 5 is a diagram showing a processing flow in the processing device 100, from acquiring a captured image from the imaging device 200, analyzing characteristic information, to outputting the result. The processing flow is mainly performed by the processor 112 of the processing device 100 reading and executing a program stored in the memory 113.

[0042] 5, before the process flow is executed, the processor 112 receives image data of the captured image of the subject captured by the imaging device 200 via the communication interface 114, and stores the image data in the image table of the memory 113 in association with the image ID information. At this time, the processor 112 also receives output values ​​measured from various sensor devices in synchronization with the imaging by the imaging device 200 via the communication interface 114, and stores the output values ​​in the image table of the memory in association with the image ID information. Furthermore, the processor 112 receives imaging condition information when the captured image was captured by the imaging device 200 via the communication interface 114, and stores the image data in the image table of the memory in association with the image ID information.

[0043] Here, an example of such a captured image is an image as shown in Fig. 1. According to Fig. 1, a pipe 11 having a hollow interior and configured to allow a fluid to pass through, a protrusion 12 formed so as to protrude from the inner peripheral surface side of the pipe 11 to the inside, and a fluid 14 passing through the inside of the pipe 11 are used as subjects. That is, the captured image includes a region in which the fluid 14 exists as a fluid region, and other regions in which the pipe 11 and the protrusion 12 and the like other than the fluid exist. In the following, a case where such a captured image is used for processing will be described.

[0044] 5, the processor 112 refers to the image table in the memory 113 and, if there is unprocessed image data, reads out the image data (S111). The processor 112 inputs the read-out image data into a trained segmentation judgment model (the creation of the trained model will be described later) and performs a fluid segmentation process (S112). At this time, only the image data is input into the trained segmentation judgment model, but in addition to this, it is also possible to further use imaging condition information, imaging environment information, or a combination thereof associated with image ID information of the captured image.

[0045] Here, FIG. 10A is a diagram showing an example of a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. Specifically, it is a diagram showing an example of segmentation image data obtained as a result of inputting image data of a captured image to the trained segmentation judgment model in S112 of FIG. 5. According to FIG. 10A, the captured image includes a region in which the fluid 14 exists as a fluid region, and a region in which the pipeline 11 and the protrusion 12, etc. other than the fluid exist as other regions, as shown in FIG. 1. By inputting the image data of such a captured image to the trained segmentation judgment model, the fluid region 24 corresponding to the region where the fluid 14 existed, and the pipeline region 21 and the protrusion region 22 corresponding to the region where something other than the fluid 14 existed are visually identified. Thus, according to the segmentation image data shown in FIG. 10A, the fluid region and other regions are identified for each pixel, and color information corresponding to the region identified for each pixel is added, so that each region is visually identified. It should be noted that visual identification using color information is used only to enable a user to identify the area when the area is output to a display or the like via the output interface 111. Therefore, it is sufficient here to specify which area each pixel (each coordinate position) belongs to, and it is of course possible to identify the area using other information, or it is not necessary to further add color information or the like. For example, the area may be identified using a grayscale, or a specific code may be added.

[0046] Returning to FIG. 5 again, the processor 112 stores the segmentation image data of the captured image obtained as a result of the processing in the image table of the memory 113 in association with the image ID information of the image data of the processed captured image (S113). Then, the processor 112 determines whether or not a fluid region exists for each pixel based on the obtained segmentation image data (S114). As a result, if it is determined that a fluid region exists, the processor 112 inputs the image data of the captured image and the segmentation image data stored in S113 into a trained fluid classification model (the creation of the trained model will be described later) and performs a classification process of the fluid for each pixel (S115). At this time, the image data and the segmentation image data are input into the trained fluid classification model, and in addition to this, imaging condition information, imaging environment information, or a combination thereof, associated with the image ID information of the captured image may be used.

[0047] Next, the processor 112 stores the classification information (water, beverage, jelly, cosmetics, medicine, food, cement, foam, edible, fuel, industrial oil, etc.) for each pixel analyzed as a result of inputting it into the learned fluid classification model for the fluid present in the fluid region specified based on the segmentation image data in the image table of the memory 113 in association with the image ID information of the image data of the processed captured image (S116). Then, the processor 112 inputs the image data of the captured image, the segmentation image data stored in S113, and the classification information stored in S116 into the learned characteristic estimation model (the creation of the learned model will be described later), and performs a characteristic estimation process of the fluid for each pixel (S117). At this time, the image data, the segmentation image data, and the classification information are inputted into the learned fluid classification model, and in addition to this, imaging condition information, imaging environment information, or a combination thereof associated with the image ID information of the captured image may be used.

[0048] Next, the processor 112 stores the characteristic information (viscosity, flow velocity, direction, velocity vector, mass, pressure, temperature or a combination thereof, etc.) estimated for each pixel as a result of inputting it into the learned characteristic estimation model for the fluid present in the fluid region identified based on the segmentation image data in an image table of the memory 113, in association with the image ID information of the image data of the processed captured image, as estimated characteristic information.

[0049] Next, the processor 112 reads out the estimated characteristic information from the memory 113 and performs post-processing to more appropriately correct the characteristic information (S118). The post-processing will be described in detail later. When the post-processing is completed, the processor 112 stores the corrected characteristic information in an image table in the memory 113 in association with the image ID information of the image data of the processed captured image (S119). A request from the user is accepted via the input interface 115, and the characteristic information is output to a display or the like via the output interface 111 based on the request (S120). This ends the processing flow.

[0050] Here, Fig. 11 is a diagram showing an example of a screen output from the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 11 is a diagram showing an example of an analysis result image output to a display via the output interface 111 in S120 of Fig. 5. According to Fig. 11, a captured image 52 captured by the imaging device 200 is displayed in the upper left of the display. In addition, an image 51 showing a velocity vector, which is one piece of corrected characteristic information, is displayed to the right of the captured image 52 so that the positional relationship between the regions can be compared. This allows the user to visually grasp the acquired characteristic information.

[0051] In addition, an imaging environment display area 55 is displayed below the captured image 52. As an example, the imaging environment display area 25 includes the pressure (atmospheric pressure) and temperature stored as imaging environment information in the image table of the memory 113, but may naturally include other imaging environment information. In addition, below the image 51 showing the velocity vector, the classification information 53 of the fluid and other characteristic information (flow velocity information 54a and viscosity information 54b) are displayed. This allows the user to visually recognize the characteristic information of the fluid analyzed by each trained model. Note that the specific information that can be displayed here is not limited to the velocity vector, flow velocity, and viscosity exemplified in FIG. 11, but may be other characteristic information such as the temperature of the fluid itself.

[0052] In addition, in FIG. 5, no particular image processing is performed on the image data of the captured image read out in S111, but it is possible to perform a predetermined pre-processing as desired. Such pre-processing includes filter processing such as band pass filters including high pass filters and low pass filters, averaging filters, Gaussian filters, Gabor filters, Canny filters, Sobel filters, Laplacian filters, median filters, and bilateral filters, trimming processing, haze removal processing, super-resolution processing, and combinations thereof, which are selected according to the purpose of high definition, region extraction, noise removal, edge enhancement, image correction, image conversion, etc. In this way, by performing pre-processing, it is possible to improve the analysis accuracy in segmentation, fluid classification, and characteristic analysis.

[0053] In addition, in the fluid segmentation process of S112 in FIG. 5, the fluid classification process of S115, and the fluid property estimation process of S117, in addition to the input image data, imaging condition information and imaging environment information may be used for processing. By using these pieces of information together, it is possible to further improve the processing accuracy. For example, the flow rate of a fluid is generally affected by the ambient temperature of the fluid or the temperature inside the fluid. Therefore, at high temperatures, the movement and change in form of the fluid are intense, and at low temperatures, they are relatively gentle, so the segmentation accuracy may be affected. In addition, even if the fluid has different flow rates under the same conditions, if one is imaged at a high temperature and the other is imaged at a low temperature, the flow rates may be similar. In such a case, the classification process and characteristic analysis process of the fluid may be affected by temperature. Therefore, a trained model trained using an image captured under high temperature conditions with a predetermined temperature as a threshold and a trained model trained using an image captured under low temperature conditions are prepared. Then, based on the imaging environment information when the input image was captured, the trained model to be input may be selected depending on whether the temperature is higher or lower than the threshold. In addition, the imaging environment information and imaging condition information can also be used as parameters to be input when each trained model is trained. For example, in the above example of temperature, since temperature can have a significant effect on the characteristics of a fluid, by also learning the temperature information, it becomes possible to output a result that takes temperature into consideration when outputting an analysis result of the characteristics from the trained model.

[0054] In addition, the image data of the captured image read out in S111 of FIG. 5 may be one image, a group of multiple still images, or a moving image composed of a group of multiple images, but it is particularly preferable to use a group of multiple still images or a moving image. In these cases, by performing the processes described in FIG. 5 on each of the read image data or a part of the image data and performing a segmentation process on the fluid region, not only can flow velocity information be obtained as characteristic information, but also time-dependent movement information and change information of the fluid region can be obtained. In particular, when analyzing flow velocity information as characteristic information of a fluid, it is preferable to use multiple still images that are different in time or moving images captured over a certain period of time. This makes it possible to more accurately reflect the change in the fluid over time, and improve the accuracy of the analysis. Furthermore, at this time, the processor 112 can calculate numerical values ​​such as the average, maximum value, and minimum value for characteristic information such as flow velocity information and viscosity information obtained from each image data. In this way, more accurate characteristic information can be obtained.

[0055] (B) Processing related to the generation of each trained model <Generating a trained segmentation decision model> Fig. 6 is a diagram showing a process flow for generating a trained model according to an embodiment of the present disclosure. Specifically, Fig. 6 is a diagram showing a process flow for generating a trained segmentation decision model used in the fluid segmentation process of S112 in Fig. 5. The process flow may be executed by the processor 112 of the processing device 100, or may be executed by a processor of another processing device.

[0056] 6, the processor executes a step of acquiring learning image data of an image captured by the imaging device 200, for example (S211). Next, the processor executes a processing step of performing a segmentation process on the acquired learning image data and assigning correct answer label information (S212). Then, the processor executes a step of storing the assigned correct answer label information as learning segmentation information in association with the learning image data (S213).

[0057] When the learning image data and the corresponding learning segmentation information are obtained, the processor executes a step of performing machine learning of a segmentation pattern using them (S214). As an example, the machine learning is performed by providing a set of information to a neural network that combines neurons, and repeating learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained segmentation judgment model is executed (S215). The acquired trained segmentation judgment model is stored in the memory 113 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0058] 6, a case has been described in which an image captured by the imaging device 200 is used as learning image data, and the learning image data is subjected to segmentation processing, for example, by a user. However, the present invention is not limited to this, and it is also possible to create a CG (computer graphics) image in which a fluid region where a fluid exists is specified in advance, and use the image data and segmentation information for learning.

[0059] In addition, in FIG. 6, a case has been described in which learning is performed using learning image data and learning segmentation information, but learning may also be performed using, for example, imaging condition information when the captured image was captured by the imaging device 200, or imaging environment information which is output values ​​measured by various sensor devices when the image was captured.

[0060] <Generating a trained fluid classification model> Fig. 7 is a diagram showing a process flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 7 is a diagram showing a process flow related to generation of a trained fluid classification model used in the fluid classification process of S115 in Fig. 5. The process flow may be executed by the processor 112 of the processing device 100, or may be executed by a processor of another processing device.

[0061] 7, the processor executes a step of acquiring learning image data of an image captured by the imaging device 200, for example, and learning segmentation information in which a fluid region is segmented using the image data (S311). Next, the processor executes a processing step of classifying the fluids present in the fluid region included in the acquired learning image data into types, and assigning correct answer label information (S312). Then, the processor executes a step of storing the assigned correct answer label information as learning classification information in association with the learning image data, etc. (S313).

[0062] When the learning image data and the corresponding learning classification information are obtained, the processor executes a step of performing machine learning of a classification pattern using them (S314). As an example, the machine learning is performed by providing a set of information to a neural network that combines neurons, and repeating learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained fluid classification model is executed (S315). The acquired trained fluid classification model is stored in the memory 113 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0063] 7, a case has been described in which a captured image captured by the imaging device 200 is used as learning image data, and the type of fluid contained in the captured image input by the user is used as the correct label information. However, this is not limiting, and it is also possible to create a CG (computer graphics) image in which a fluid region where a fluid exists and the type of the fluid are specified in advance, and use this as learning image data, learning segmentation information, and learning classification information.

[0064] In addition, in FIG. 7, a case has been described in which learning is performed using learning image data, learning segmentation information, and learning classification information. However, learning may also be performed using, for example, imaging condition information when the captured image was captured by the imaging device 200, or imaging environment information which is output values ​​measured by various sensor devices when the image was captured.

[0065] <Generation of trained characteristic estimation model> Fig. 8 is a diagram showing a process flow related to generation of a trained model according to an embodiment of the present disclosure. Specifically, Fig. 8 is a diagram showing a process flow related to generation of a trained property estimation model used in the fluid property analysis process of S117 in Fig. 5. The process flow may be executed by the processor 112 of the processing device 100, or may be executed by a processor of another processing device.

[0066] 8, the processor executes a step of acquiring learning image data of an image captured by the imaging device 200, learning segmentation information in which a fluid region is segmented using the image data, and classification information in which the type of fluid present in the fluid region is identified (S411). Next, the processor executes a processing step of analyzing the characteristics of the fluid present in the fluid region in the acquired learning image data and assigning correct label information (S412). Then, the processor executes a step of storing the assigned correct label information as learning characteristic information in association with the learning image data, etc. (S413).

[0067] When the learning image data and the like and the learning characteristic information associated therewith are obtained, the processor executes a step of performing machine learning of the characteristic analysis pattern using them (S414). As an example, the machine learning is performed by providing a set of the information to a neural network that combines neurons, and repeating learning while adjusting the parameters of each neuron so that the output from the neural network is the same as the correct label information. Then, a step of acquiring a trained characteristic estimation model is executed (S415). The acquired trained characteristic estimation model is stored in the memory 113 of the processing device 100 or in another processing device connected to the processing device 100 via a wired or wireless network.

[0068] Here, the Navier-Stokes equations shown in the following equation 1 are used as equations that describe the behavior of a fluid. In other words, in order to be a fluid, each piece of characteristic information analyzed using the trained characteristic estimation model or the like must satisfy the Navier-Stokes equations shown in the following equation 1.

[0069]

number

[0070] Therefore, in the training of the trained characteristic estimation model, the loss function applying the Navier-Stokes equation as shown in the above formula 1 is used to calculate the loss (L n In other words, the parameters of each neuron are adjusted so that, for each piece of characteristic information output from the trained characteristic estimation model, the parameters of each neuron are adjusted so that, when input to the Navier-Stokes equation of the above formula 1, the value satisfies the equation.

[0071] In the above formula 1, the pressure (p), acceleration field (g), and fluid density (ρ) can be ignored as fixed values. As the velocity vector (υ), flow velocity information obtained from the trained characteristic estimation model is used, and as the dynamic viscosity coefficient (ν), viscosity information obtained from the trained characteristic estimation model is used.

[0072] 8, a case has been described in which an image captured by the imaging device 200 is used as learning image data, and the characteristics of the fluid measured by the user are used as the correct label information. However, this is not limiting, and it is also possible to create a CG (computer graphics) image in which the fluid region where the fluid exists and the type and characteristics of the fluid are specified in advance, and use this as the learning image data, learning segmentation information, learning classification information, and learning characteristic information.

[0073] 6 to 8, the case where learning is performed using the learning image data, learning segmentation information, learning classification information, and learning characteristic information has been described, but learning may be performed using, for example, imaging condition information when the captured image is captured by the imaging device 200, or imaging environment information that is output values ​​measured by various sensor devices when the image is captured. By using this information, it becomes possible to output a result that further considers the imaging environment information and imaging condition information when outputting an analysis result of the characteristic from the trained model.

[0074] Further, each trained model described in FIG. 6 to FIG. 8 was generated using a neural network. However, the trained models are not limited to these, and can be generated using machine learning such as a nearest neighbor method, a decision tree, a regression tree, and a random forest. Further, each trained model described in FIG. 6 to FIG. 8 was generated using a neural network. Specific examples include machine learning models such as FCN and Mask R-CNN. However, the trained models are not limited to these, and any machine learning model that can input an image or output pixel data can be used.

[0075] (C) Processing related to correction of characteristic information Fig. 9 is a diagram showing a processing flow executed in the processing device 100 according to an embodiment of the present disclosure. Specifically, Fig. 9 is a diagram showing a processing flow of post-processing performed in S118 of Fig. 5, i.e., processing related to correction of estimated characteristic information. This processing flow is mainly performed by the processor 112 of the processing device 100 reading and executing a program stored in the memory 113. Note that, for convenience of explanation, the following will be described in the case where a velocity vector is estimated as characteristic information.

[0076] As shown in FIG. 9, the processor 112 reads out the velocity vector, which is the characteristic information estimated in S117, from the memory 113 to obtain the result of the estimation process for each pixel (S211).

[0077] Here, as shown in FIG. 1, when the fluid 14 moves constantly from left to right inside the pipe 11, it is possible to appropriately estimate the characteristic information of the fluid in the characteristic estimation process of S117. However, in reality, there are often factors such as protrusions 12 that hinder the movement of the fluid 14 in the space through which the fluid 14 moves. When such a protrusion 12 exists, for example, in the region 13a in front of the protrusion 12, the flow speed of the fluid 14 temporarily slows down and the flow direction also changes along the shape of the protrusion 12. In addition, after passing the protrusion 12, the speed temporarily increases in the vicinity of the end of the protrusion 12 with respect to the direction along the pipe, and the fluid moves significantly, while in the region 13b near the rising edge of the protrusion 12, the fluid exhibits a swirling behavior. Therefore, compared to the case where the fluid 14 moves constantly and constantly, the characteristic information also changes in a complex manner in the region 13b. For example, in the region 13a in front of the protrusion 12, the flow speed decreases, while the flow direction changes upward along the protrusion 12. In the region 13b near the rising edge of the protrusion 12, the flow velocity decreases compared to the upper portion, while the direction of the flow changes in a complex manner like a vortex. For the sake of convenience, the following description will be given using a velocity vector as an example of characteristic information, but the present invention is not limited to this.

[0078] FIG. 10B is a diagram showing an example of characteristic information generated from a segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 10B shows characteristic information estimated in S117 of FIG. 5 and acquired in S211 of FIG. 9. According to FIG. 10B, a conduit region 31 and a protrusion region 32 are shown corresponding to the conduit region 21 and the protrusion region 22 of FIG. 10A. In addition to these, a velocity vector, which is characteristic information estimated for each pixel, is shown by an arrow. The length of the arrow indicates the flow velocity (the longer the arrow, the faster the flow velocity), and the direction of the flow of the fluid is indicated by the direction indicated by the arrowhead. Here, as described in FIG. 1, it is shown that the arrow indicating the velocity vector moves significantly along the movement direction of the fluid in the conduit region 31 (from left to right in FIG. 10B) near the region 34 that is beyond the end of the protrusion region 32 and along the conduit region 31.

[0079] However, near region 33a just before protrusion region 32, the flow speed temporarily slows down, causing the arrows indicating the velocity vectors to become excessively small, making it appear as if the fluid is barely moving. Also, near region 33b near the rising edge of protrusion region 32, the fluid that would normally change in a spiral shape is now pointing in a direction completely different from the velocity vectors of the surrounding pixels, or is shown as being excessively large or small compared to the magnitude of the velocity vectors of the adjacent pixels.

[0080] Therefore, when there is a factor that hinders the movement of the fluid (e.g., protrusion 12 in Figure 1), it is desirable to correct the estimated characteristic information to obtain more accurate specific information.

[0081] Returning to FIG. 9 again, when the velocity vector for each pixel is read, the processor 112 estimates the reliability of the velocity vector for each pixel (S212). Here, the "reliability" is one of the indices indicating the accuracy of the characteristic information estimated in S118 of FIG. 5. As described in FIG. 1, the characteristic information may not be correctly estimated around the protrusion 12. This is because the flow velocity is locally reduced in the region 13a and the region 13b around the protrusion 12, so that the time change of the pixel is small. Furthermore, when using a trained model as described below, if a training image including an obstacle such as the protrusion 12 is used, it is difficult to properly train the above-mentioned location where the time change of the pixel is small. In this way, it is necessary to identify pixels estimated to have low reliability as a result of the estimation of the velocity vector, and correct the velocity vector of the pixel.

[0082] As an example, the reliability will be described in the case where the image data of the captured image includes at least the frame image data of the tth frame captured during a predetermined period (hereinafter referred to as "t frame image data") and the frame image data of the t+1th frame captured after an arbitrary time has elapsed since the t frame image data (hereinafter referred to as "t+1 frame image data"). First, the processor 112 analyzes each pixel of the t frame image data of the captured image to extract one or more feature points contained in the image data. Similarly, the processor 112 analyzes each pixel of the t+1 frame image data to extract one or more feature points contained in the image data. Then, the processor 112 calculates a first index which is the amount of movement of each of the one or more feature points extracted from the t frame image data in the t+1 frame image data. For example, the larger the value of the first index, the larger the amount of movement. Note that such a calculation method of the first index is merely one example of a feature point matching method. Any feature point matching method may be used as long as it is possible to calculate the amount of movement for each pixel as the first index.

[0083] Next, the processor 112 reads out the velocity vector (estimated value) of each pixel obtained by inputting the t frame image data into the trained characteristic analysis model. The processor 112 also calculates the velocity vector (actual value) of each pixel of the t frame image data by comparing the t frame image data with the t+1 frame image data. The processor 112 then applies the estimated value of the velocity vector and the actual value of the velocity vector to a loss function for each pixel, and calculates a second index, which is a loss value for each pixel. For example, the larger the value of the second index, the greater the loss, that is, the greater the deviation between the estimated value and the actual value.

[0084] Next, the processor 112 calculates the reliability (in this case, the reliability is a value between 0 and 1) by dividing the calculated second index by the first index. That is, the reliability is relatively expressed as follows: If the first index is large (the amount of movement is large) and the second index is small (the deviation between the estimated value and the actual measured value is small), the reliability is "high" (for example, near the area 34 in FIG. 10B). When the first index is large (the amount of movement is large) and the second index is large (the deviation between the estimated value and the actual measured value is large), the reliability is lower than "high" and higher than "low" (for example, near the area 33b in FIG. 10B). When the first index is small (the amount of movement is small) and the second index is small (the deviation between the estimated value and the actual measured value is small), the reliability is lower than "high" and higher than "low" (for example, near the area 33b in FIG. 10B). When the first index is small (the amount of movement is small) and the second index is large (the deviation between the estimated value and the actual measured value is large), the reliability is "low" (for example, near the area 33a in FIG. 10B). This value indicates:

[0085] Such reliability may be any one that indicates the likelihood of the characteristic information, and may be, for example, only one of the first index and the second index, or may be combined with or used instead of another index. In addition, although the reliability uses a specific value obtained by dividing the second index by the first index in the above, any classification or parameter obtained based on the value may be used.

[0086] Next, the processor 112 corrects the acquired characteristic information based on the estimated reliability of each pixel, and generates corrected characteristic information (S213). As an example, the correction is performed by multiplying the characteristic information of a pixel with a larger correction coefficient for a pixel with a lower reliability. The gradient of the velocity vector field of each pixel is used as the correction coefficient. That is, the correction is calculated by processing the following calculation formula (Formula 2).

[0087]

number

[0088] According to this formula, the higher the reliability (value between 0 and 1) calculated for each pixel, the smaller the correction coefficient by which the estimated velocity vector is multiplied, and the lower the reliability, the larger the correction coefficient by which the estimated velocity vector is multiplied. Therefore, the amount of correction applied to the velocity vector of a pixel with low reliability is greater.

[0089] The above correction is merely an example of a process or method for correction. For example, a table in which reliability and correction coefficients are associated with each other may be prepared in advance, and the correction coefficients specified by the table may be multiplied. In addition to the correction coefficient and reliability as described above, other parameters may also be multiplied.

[0090] Here, FIG. 10C is a diagram showing an example of correcting the characteristic information generated from the segmentation image processed by the processing device 100 according to an embodiment of the present disclosure. Specifically, FIG. 10C shows the characteristic information after correction performed in S213 of FIG. 9. According to FIG. 10C, similar to FIG. 10B, the conduit region 31 and the protrusion region 32 are shown corresponding to the conduit region 21 and the protrusion region 22 of FIG. 10A. In addition to these, the velocity vector, which is characteristic information estimated for each pixel, is shown by an arrow. In FIG. 10B showing the characteristic information before correction, in the region 33a and the region 33b where the movement amount of the feature point between the t frame and the t+1 frame is small and therefore the velocity vector could not be accurately calculated, FIG. 10C shows that the velocity vector has been corrected to a larger value by the correction process. For example, in the vicinity of the region 33a in front of the protrusion region 32, the change in flow speed and direction due to the collision of the fluid with the wall of the protrusion region 32 is more accurately shown. Moreover, in the vicinity of region 33b near the rising edge of protruding region 32, the state of the fluid changing in a swirling manner is more accurately shown.

[0091] Returning to FIG. 9 again, when the processor 112 generates the corrected characteristic information as described above, it ends the post-processing and stores the corrected characteristic information, which has been corrected in accordance with S119 of FIG.

[0092] As described above, in one embodiment of the present disclosure, a processing device, a processing program, and a processing method suitable for analyzing characteristic information of a fluid can be provided. In particular, by calculating the reliability for each pixel and correcting the estimated characteristic information using a larger correction coefficient for pixels with low reliability, more accurate characteristic information can be obtained.

[0093] The processes and procedures described in this specification can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination of these. Specifically, the processes and procedures described in this specification can be realized by implementing logic corresponding to the processes in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, a magnetic disk, or an optical storage. In addition, the processes and procedures described in this specification can be implemented as computer programs and executed by various computers including a processing device and a server device.

[0094] Although the processes and procedures described herein are described as being executed by a single device, software, component, or module, such processes or procedures may be executed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, although the various information described herein is described as being stored in a single memory or storage unit, such information may be stored in multiple memories provided in a single device or multiple memories distributed across multiple devices. Furthermore, the software and hardware elements described herein may be realized by integrating them into fewer components or breaking them down into more components. [Explanation of symbols]

[0095] 1 Processing System 100 Processing equipment 200 Imaging device

Claims

1. A processing device comprising at least one processor, the at least one processor: acquiring image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; The acquired image data is input to a trained characteristic estimation model for generating characteristic information indicating at least one of a speed and a direction of a fluid, thereby estimating the characteristic information for each of one or more pixels constituting the image data; calculating an index indicating a reliability of the estimated characteristic information for each of the one or more pixels; correcting the characteristic information of the one or more pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability; A processing device configured to process the

2. The processing device of claim 1 , wherein the characteristic information is a velocity vector indicating a velocity and direction of the fluid.

3. The processing device according to claim 1 , wherein the learned characteristic estimation model is generated by machine learning using at least image data of the learning image and learning characteristic information calculated for one or more pixels in the image data of the learning image.

4. The processing device according to claim 1 , wherein the index is calculated by calculating an amount of movement of an object included in the image data for each of the one or more pixels over a predetermined period of time, and indicates that the reliability is higher as the amount of movement is larger.

5. The processing device according to claim 1 , wherein the index indicates that the reliability is higher as the estimation accuracy of the characteristic information is higher for each of the one or more pixels.

6. The processing device according to claim 1 , wherein the index indicates that, for each of the one or more pixels, the reliability is higher as an amount of movement of the characteristic information in a predetermined period is larger and an estimation accuracy of the characteristic information is higher.

7. The processing device according to claim 1 , wherein the correction is performed based on a correction coefficient that varies depending on the reliability.

8. 2. The treatment device according to claim 1, wherein the fluid is at least one of a gas, a liquid, a mixture of a gas and a liquid, and a mixture of a gas or a liquid with a trace amount of a solid.

9. When executed by at least one processor in the processing device, The at least one processor, acquiring image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; The acquired image data is input to a trained characteristic estimation model for generating characteristic information indicating at least one of a speed and a direction of a fluid, thereby estimating the characteristic information for each of one or more pixels constituting the image data; calculating an index indicating a reliability of the estimated characteristic information for each of the one or more pixels; correcting the characteristic information of the one or more pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability; A processing program that makes it function as such.

10. A processing method executed by at least one processor included in a processing device, comprising: acquiring image data of an image captured by an imaging device included in the processing device or connected to the processing device via a communication interface; A step of estimating the characteristic information for each one or more pixels constituting the image data by inputting the acquired image data into a learned characteristic estimation model for generating characteristic information indicating at least one of a velocity and a direction of a fluid; calculating an index indicating a reliability of the estimated characteristic information for each of the one or more pixels; correcting characteristic information of the one or more pixels based on characteristic information generated in one or more pixels different from the one or more pixels based on the calculated reliability; A processing method comprising:

Images