Information processing system, program, and information processing method

Abstract

To provide an information processing system, an information processing method, and an information processing program which can reduce a load of a server and improve diagnostic accuracy by reduction of a data quantity by accepting only image data with a target object included in a field angle and performing identification and labelling of only data obtained by excluding data with an inappropriate exposure state, etc.SOLUTION: In an image processing system with a plurality of image pickup devices, a plurality of image collection devices, and a server connected with one another through a network, an image processing method includes: accepting image data obtained by the image pickup devices S101; recognizing a target object included in the image data S102; extracting only the image data with the target object accommodated in a field angle S103; excluding the image data with inappropriate exposure and focus among the extracted images S105; fetching the image data after exclusion S106; performing identification and labeling S107; and notifying a user of labeling information after identification S108.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a program, and an information processing method. [Background technology]

[0002] The deterioration of various internal engine parts, such as piston rings in internal combustion engines, cannot be seen from the outside. Therefore, it is common to insert a camera or other device inside the engine and check the condition from the image information obtained. Furthermore, this inspection often requires the experience of a skilled worker, and is not something that an amateur can easily judge.

[0003] Patent Document 1 describes a dental analysis system that can assist dentists by diagnosing lesions and the like using machine learning on dental panoramic X-ray image data obtained by capturing images with an X-ray camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-208831 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as disclosed in Patent Document 1, if the server itself were to receive all the X-ray image data, the amount of data and maintenance costs for the server would increase, and the load would also increase. Furthermore, data with improper camera focus, etc. would also be accumulated as training data, which would affect the overall diagnostic accuracy.

[0006] In view of the above circumstances, the present invention aims to provide an information processing system, program, and information processing method that can reduce the load on the server by reducing the amount of data and improve diagnostic accuracy by accepting only image data obtained by an image capturing device in which the target object is included in the angle of view, and further excluding data with inappropriate exposure conditions, etc., and identifying and labeling only the data. [Means for solving the problem]

[0007] A first aspect of the present invention is An information processing system, It is configured to perform the following steps: In the receiving step, image data obtained by the image capturing device is received, In the extraction step, a target object included in the image data is recognized, and only the image data in which the target object is included in an angle of view is extracted; In the excluding step, image data having inappropriate exposure and focus is excluded from the extracted images; In the identification step, the image data after the exclusion is identified; In the notification step, the user is notified of the labeling information after the identification. death, The target object is a piston ring or a ring land inside a cylinder of a marine diesel engine, In the excluding step, an image in which an edge of soot adhering to the piston ring or ring land cannot be extracted is determined to be an image with an improper focus. , is something.

[0008] A second aspect of the present invention is An information processing method, comprising: It includes the following steps: In the receiving step, image data obtained by the image capturing device is received, In the extraction step, a target object included in the image data is recognized, and only the image data in which the target object is included in an angle of view is extracted; In the excluding step, image data having inappropriate exposure and focus is excluded from the extracted images; In the identification step, the image data after the exclusion is identified; In the notification step, the user is notified of the labeling information after the identification. death, The target object is a piston ring or a ring land inside a cylinder of a marine diesel engine, In the excluding step, an image in which an edge of soot adhering to the piston ring or ring land cannot be extracted is determined to be an image with an improper focus. , method.

[0009] According to this, only image data obtained by the image capturing device that includes the target object within the angle of view is accepted, and only data with inappropriate exposure conditions, etc. is identified and labeled, thereby reducing the load on the server by reducing the amount of data and improving identification accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is an overall diagram of an information processing system according to an embodiment of the present invention; [Figure 2(a)] FIG. 2 is a block diagram showing the hardware configuration of the image collection device 3. [Figure 2(b)] FIG. 2 is a block diagram showing the hardware configuration of the server 4. [Figure 3(a)] 1 is a block diagram showing functions realized by an image collection device 3. [Figure 3(b)] FIG. 2 is a block diagram showing functions realized by a server 4. [Figure 4] FIG. 2 is a flowchart of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings. Various features shown in the following embodiments can be combined with each other.

[0012] 1. Hardware Configuration In Section 1, the hardware configuration of this embodiment will be described.

[0013] 1.1 Information Processing System 1 1 is an overall diagram of an information processing system according to this embodiment. The information processing system 1 includes an image capturing device 2 (e.g., image capturing devices 2-1, 2-2, . . . , 2-n), an image collecting device 3 (e.g., image collecting devices 3-1, 3-2, . . . , 3-n), and a server 4, which are connected via a network. These components will be further described. Here, a system exemplified as the information processing system 1 is made up of one or more devices or components.

[0014] 1.2 Image capturing device 2 To facilitate subsequent identification and labeling of the target object, it is preferable that multiple image capture devices 2 be installed in a given space to avoid blind spots. Furthermore, if the user can enter the target object, the image capture device 2 may be manually operated by the user. It may also be a single image capture device. The image capture device 2 may be, for example, a digital camera or digital video camera. A 4K (3840 x 2160 pixel) camera is preferable for capturing a wide area while maintaining high image resolution. To reduce the amount of data transmitted to the image collection device 3, still images are preferable for the image data transmitted to the image collection device 3. For video, a frame rate of 30 FPS (frames per second) or less is preferable. A frame rate of 5 FPS or less is more preferable, specifically, 5, 10, 15, 20, 25, or 30 FPS, or any two of the values ​​exemplified here. Lowering the frame rate can reduce the amount of data transmitted to the image collection device 3 (described below), thereby reducing the load when performing the extraction step.

[0015] The image capturing device 2 is connected to a communication unit 31 in the image collecting device 3 (described later) via a network. The image capturing device 2 is configured to transfer all captured image data to the image collecting device 3.

[0016] Furthermore, an imaging device capable of measuring not only visible light but also wavelengths imperceptible to humans, such as ultraviolet and infrared regions, may be employed as the imaging device 2. By employing such an imaging device 2, the information processing system 1 according to this embodiment can be implemented even in a dark field.

[0017] 1.3 Image acquisition device 3 The image collection device 3 is provided corresponding to each image capture device 2, and has the same number as the number associated with the image capture device 2. Note that when one of the image capture devices 2 captures the target object in its field of view, one image collection device 3 may be provided corresponding to a plurality of image capture devices 2 so that other image capture devices 2 can be linked to facilitate identification and labeling of the target object. The number of image capture devices 2 connected to one of the image collection devices 3 is not particularly specified as long as it can accommodate the amount of data transmitted from the image capture devices 2. The image collection device 3 is a terminal device, such as an edge device.

[0018] 2(a) is a block diagram showing the hardware configuration of the image collecting device 3. The image collecting device 3 has a communication unit 31, a storage unit 32, and a control unit 33, and these components are electrically connected via a communication bus 30 inside the image collecting device 3. Each component will be further described.

[0019] The communication unit 31 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt, or wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 3G, Bluetooth (registered trademark) communication, etc. as needed. In other words, it is more preferable to implement it as a collection of multiple communication means. The image collection device 3 receives image data from the image capturing device 2 and transmits only image data in which the target object is included in the angle of view to the server 4 via the communication unit 31. Details will be described later.

[0020] The storage unit 32 temporarily stores the image data received from the image capturing device 2. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs related to the image collecting device 3 executed by the control unit 33, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the program's calculations. It may also be a combination of these. In particular, the image data is stored until the "extraction of image data of the target object" described in step S103 of FIG. 4 is performed and the "deletion from memory, etc." described in step S104 of FIG. 4 is performed.

[0021] The control unit 33 processes and controls the overall operations related to the image collection device 3. The control unit 33 is, for example, a central processing unit (CPU) not shown. The control unit 33 recognizes the target object by reading a predetermined program stored in the storage unit 32, and extracts only image data in which the target object is included in the angle of view. In other words, information processing by software stored in the storage unit 32 is specifically realized by the control unit 33, which is an example of hardware, and can be executed as a functional unit included in the control unit 33. These will be described in more detail in the next section. Note that the control unit 33 is not limited to being single, and multiple control units 33 may be provided for each function. A combination of these may also be used.

[0022] 1.4 Server 4 2(b) is a block diagram showing the hardware configuration of the server 4. Like the image collection device 3 described above, the server 4 has a communication unit 41, a storage unit 42, and a control unit 43, and these components are electrically connected inside the server 4 via a communication bus 40. Therefore, only the parts of each component that are different from those described above will be described.

[0023] The server 4 receives image data in which the target object is contained within the angle of view from the image collection device 3 via the communication unit 41. Note that the server 4 may also be connected to the image capture device 2 in order to determine which portion of all the image data transmitted from the image capture device 2 to the image collection device 3 corresponds to the image data. Details will be described later.

[0024] The control unit 43 reads out a predetermined program stored in the storage unit 42, thereby excluding image data with inappropriate exposure and focus as described in step S105 of Fig. 4, and the storage unit 42 stores only the excluded image data as data to be analyzed. In particular, the data is stored until "save the data to be analyzed as training data" as described in step S110 of Fig. 4 is executed, after which the data to be analyzed is stored as training data in another area of ​​the storage unit 42.

[0025] The control unit 43 realizes the identification and labeling of the target object as described in step S107 of Fig. 4 by reading out a predetermined program stored in the storage unit 42. After that, the data for which the identification and labeling have been completed is temporarily stored in a separate area of ​​the storage unit 42.

[0026] After the above identification and labeling are completed, the user is notified of the completion, and the labeled data temporarily stored in the storage unit 42 is updated and generated when "save the data to be analyzed as training data" as described in step S110 of Fig. 4 is executed. Note that the training data can be updated and generated only for a portion of the data, and the user may be able to select and discard the data.

[0027] In the case where the extraction step is called the first screening and the exclusion step is called the second screening, the first screening performed by the image collection device 3 may also be performed by the server 4. By reading out a predetermined program stored in the storage unit 42, both screenings may be performed simultaneously, and various combinations are possible depending on the number of image capturing devices 2, the data capacity of the server 4, etc.

[0028] 2. Functional configuration This section describes the functional configuration of this embodiment. As described above, information processing by software stored in the storage units 32 and 42 is specifically realized by the control units 33 and 43, which are an example of hardware, and can be executed as functional units included in the control units 33 and 43.

[0029] 3(a) is a block diagram showing functions realized by the image collecting device 3 (control unit 33). Specifically, the image collecting device 3 (control unit 33) includes a receiving unit 331, a recognizing unit 332, an extracting unit 333, and a transmitting unit 334.

[0030] The receiving unit 331 is configured to receive various types of information. For example, the receiving unit 331 receives all image data sent from the image capturing device 2.

[0031] The recognition unit 332 is configured to automatically recognize whether a target object is contained within the angle of view of all image data transmitted from the image capturing device 2. For example, the recognition unit 332 calls a predetermined program stored in the storage unit 32, extracts features (edge ​​extraction) from all image data transmitted from the image capturing device 2, analyzes them, and automatically recognizes the target object. It is more preferable that the automatic recognition is performed by machine learning based on training data stored in the server 4 and sent to the image collection device 3, and the feature extraction algorithm may be included as necessary. There are no particular limitations on the machine learning algorithm, and it is sufficient to appropriately adopt a k-nearest neighbor method, logistic regression, support vector machine, neural network, topic model, Gaussian mixture model, etc.

[0032] The extraction unit 333 is configured to identify an image in which the target object is contained within the field of view through the automatic recognition and extract only that image. If the entire target object is not contained within the field of view, it is preferable to connect the image in a specific direction to create a new extracted image. This is because it reduces the amount of data sent to the server 4, reduces the load, and improves the accuracy of the information processing system 1.

[0033] At this time, images that have not been extracted are automatically deleted from the memory or the like provided in the image collection device 3. Furthermore, if one image collection device 3 is connected to multiple image capturing devices 2, the extraction of the images may be performed in conjunction with each other, and the above-mentioned image linking and extraction may be performed.

[0034] The transmission unit 334 is configured to transmit the extracted image data to the server 4. Note that, when one image collection device 3 is connected to a plurality of image capturing devices 2, the transmission unit 334 may be configured to check the linkage, i.e., whether or not some images of the image capturing devices 2 are missing, and transmit the images to the server 4 after the above-mentioned image linking and extraction.

[0035] 3(b) is a block diagram showing functions realized by the server 4 (control unit 43). Specifically, the server 4 (control unit 43) includes a receiving unit 431, an exclusion unit 432, an identification unit 433, and a transmission unit 434.

[0036] The receiving unit 431 is configured to receive various types of information. For example, the receiving unit 431 receives image data from the image collection device 3 in which the target object is contained within the angle of view.

[0037] The exclusion unit 432 excludes image data with an inappropriate depth of focus or poor exposure from the data in which the target object is included in the angle of view. Image data with an inappropriate depth of focus refers to image data in which the target object is included in the angle of view but from which the edges of other objects that may be present in association with the target object cannot be extracted. For example, if the target object is a piston ring of a cylinder in an internal combustion engine, soot may be an example of such an object that may be present in association with the target object. Similarly, image data with poor exposure refers to image data captured at an illumination level at which the presence of soot or the like cannot be confirmed.

[0038] After the exclusion by the exclusion unit 432, the identification unit 433 reads a predetermined program stored in the storage unit 42, identifies the state of the target object, and performs classification and labeling. Thereafter, the transmission unit 434 notifies the user that the identified and labeled information will be stored in the storage unit 42 of the server 4. The user may decide whether to store each or all of the data as training data, and all of the data may be stored as training data after a predetermined time has elapsed.

[0039] 3. Information Processing Method This section describes an information processing method of the information processing system 1 described above. This information processing method includes the following steps: In the receiving step, image data obtained by an image capturing device is received; In the extracting step, a target object contained in the image data is recognized, and only the image data in which the target object is contained within the angle of view is extracted; In the excluding step, image data with inappropriate exposure and focus is excluded from the extracted images; In the identifying step, the image data after the excluding is identified; and In the notifying step, the user is notified of the labeling information after the identification.

[0040] 4 is a flowchart showing the flow of information processing executed by the information processing system 1. Below, a case where the target object is a piston ring of a cylinder in an internal combustion engine will be described along each flow of this flowchart.

[0041] When a user wishes to inspect the condition of a piston ring of a cylinder in an internal combustion engine, the user starts the image capturing device 2. As described above, the user may manually start and capture images by operating the image capturing device 2, and the image collecting device 3 is configured to accept all image data from the image capturing device 2 (step S101). The image collecting device 3 automatically recognizes whether the piston ring and ring land, which are the target objects, are included in the angle of view of the image data received from the image capturing device 2 (step S102). As described above, the automatic recognition may be performed using an edge extraction method for extracting feature amounts or machine learning using supervised data. Furthermore, a combination of these may also be used.

[0042] When the machine learning using the supervised data uses a convolutional neural network, it is possible to directly learn features, which is particularly preferable in that it can partially eliminate the effort required for the user to extract features and improve the accuracy of the information processing system 1.

[0043] After the automatic recognition is performed, the image collecting device 3 extracts only image data in which the piston ring and the ring land are included in the angle of view (step S103), and transmits the extracted image data to the server 4. A so-called primary screening is performed by the image collecting device 3. If the entire piston ring and the ring land are not included in the angle of view, the images may be linked in a specific direction and transmitted to the server 4 as a new extracted image.

[0044] On the other hand, image data that is not entirely contained within the angle of view is erased (step S104) from a storage device such as a solid state drive (SSD) or a memory such as a random access memory (RAM) provided in the image collection device 3. The data erased in step S104 is all data that was not erased in step S103, and since the image collection device 3 accepts all image data from the image capturing device 2, it is preferable that this be executed as soon as possible after step S103.

[0045] Image data sent to the server 4 that has an inappropriate exposure state or depth of focus is excluded. A so-called secondary screening is performed by the server 4 (step S105). In particular, in marine diesel engines, soot is generated inside the cylinder due to the fuel. As described above, if the exposure state or depth of focus is inappropriate, unwanted substances such as soot adhering to the piston rings and ring lands cannot be identified from the image. The above-mentioned method of determining the data to be excluded by automatic recognition includes edge extraction, which extracts feature quantities of unwanted substances such as soot, and machine learning using supervised data. A combination of these methods may also be used.

[0046] If the machine learning using the supervised data uses a convolutional neural network separate from the primary screening, the features can be learned directly, which is particularly preferable in that it can partially eliminate the effort required for the user to extract features and improve the accuracy of the information processing system 1.

[0047] The determination of the data to be excluded is carried out by reading out a predetermined program stored in the memory unit 42 of the server 4 as described above, and it goes without saying that various parameters etc. are adjusted according to the amount of supervised data described below.

[0048] Only the image data that has been excluded is imported as the subject of analysis (step S106). A detailed assessment is made of microsieves, scuffing, damage, degree of wear of the coating, angle cut, S-lock state, etc., and the state is identified, classified, and labeled (step S107). The identification and labeling through this analysis is performed by reading out a predetermined program stored in the memory unit 42 of the server 4 as described above, and it goes without saying that various parameters are adjusted depending on the user's classification accuracy and classification purpose. This analysis mainly involves clustering using multivariate analysis, and the parameters are so-called weighting values, etc.

[0049] The above is for piston rings, but in the case of ring lands, the top, first, second, and third ring lands are each classified into normal, disclosure, light carbon, and excessive conditions.

[0050] If the classification and labeling are performed by machine learning using supervised data, as described above, and a convolutional neural network separate from the primary and secondary screening is used, this is particularly preferable in that it allows features to be learned directly, thereby improving the accuracy of the information processing system 1.

[0051] Thereafter, the user is notified that the above-mentioned identification and labeling have been completed (step S108). The user receives the notification and decides whether or not to save some or all of the identified and labeled data as training data (step S109). The training data is then saved in the server 4 (step S110), and the training data in the image collecting device 3 is updated to contribute to improving the accuracy of automatic recognition in the image collecting device 3 (step S111). The user is then notified that the training data in the image collecting device 3 has been updated, thereby completing the series of steps (step S112).

[0052] Note that a so-called loop-type information processing method may be used in which the accuracy of the primary screening is improved using the teacher data updated in step S111. In the determination of step S109, it may be left to the user's discretion as to whether to loop again.

[0053] 4.Other The information processing system 1 according to this embodiment may be configured as follows: In the information processing system 1, a part of the image data extracted in the extraction step is connected in a specific direction to generate a new extracted image in which the entire target object is included in the angle of view. This is preferable because it not only reduces the amount of data sent to the server 4 and reduces the load, but also improves the accuracy of the information processing system 1.

[0054] In the information processing system 1, the extraction of image data in the extraction step, the exclusion of image data in the exclusion step, and the identification of image data in the identification step are based on learned data that has been learned in advance. Using machine learning is more preferable because it enables the information processing system 1 to perform highly accurate identification and classification.

[0055] In the information processing system 1 using the above-mentioned machine learning, the server 4 and the image collection device 3 further include the learning step, and generate or update the learned data using the image data after the identification step is completed as training data. As described in steps S110 and S111 of FIG. 4, the actual data after classification is updated as training data, which is more preferable since it enables the information processing system 1 to perform classification and classification with higher accuracy.

[0056] In the information processing system 1, the target object is a piston ring inside a cylinder of a marine diesel engine. The condition of the piston ring often requires visual judgment by a skilled worker, but since various parameters, particularly scuffing, are difficult to quantify, it is preferable to be able to make this judgment automatically. Furthermore, it is even more preferable that this automatic judgment (automatic identification) be made by machine learning based on supervised data, as this is expected to improve accuracy.

[0057] The information processing system 1 is configured to classify the identification data. As mentioned above, it is difficult to express the condition of piston rings and the like inside a cylinder in absolute numerical value, but classifying and labeling the identification results allows the user to better determine the condition. This makes it easier to select what to do in step S109, which is preferable because it improves the accuracy of the information processing system 1.

[0058] Although the above embodiment has been described as a configuration of the information processing system 1, a program that causes a computer to execute each step in the information processing system 1 may be provided. Even if the image collection device 3 does not have a sufficient data capacity, it is preferable because the server 4 can start up a plurality of programs (steps S103, S105, S107, etc.) and achieve the object of the present invention.

[0059] Finally, while various embodiments of the present invention have been described, these are presented by way of example only and are not intended to limit the scope of the invention. The novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. Such embodiments and modifications are intended to be included within the scope and spirit of the invention, as well as within the scope of the inventions and their equivalents as defined in the accompanying claims. [Explanation of symbols]

[0060] 1: Information processing system 2: Image capture device 3: Image acquisition device 30: Communication bus 31: Communications Department 32: Storage section 33: Control section 331: Reception 332: Recognition part 333:Extraction part 334: Transmission unit 4: Server 40: Communication bus 41: Communications Department 42: Storage section 43: Control section 431: Reception 432: Exclusion part 433: Identification unit 434: Transmission unit

Claims

1. An information processing system, It is configured to perform the following steps: In the receiving step, image data obtained by the image capturing device is received, In the extraction step, a target object included in the image data is recognized, and only the image data in which the target object is included in an angle of view is extracted; In the excluding step, image data having inappropriate exposure and focus is excluded from the extracted images; In the identification step, the image data after the exclusion is identified; In the notification step, a user is notified of the identified labeling information; The target object is a piston ring or a ring land inside a cylinder of a marine diesel engine, In the excluding step, an image in which an edge of soot adhering to the piston ring or ring land cannot be extracted is determined to be an image with an improper focus.

2. 2. The information processing system according to claim 1, Extracting image data in the extracting step; excluding image data in the excluding step; Identification of image data in the identifying step However, it is based on pre-trained data.

3. 3. The information processing system according to claim 2, The information processing system further comprises a learning step, In the learning step, the image data after classification is used as training data to generate or update the learned data.

4. In the information processing system according to any one of claims 1 to 3, The identifying step is configured to classify the state of the target object.

5. An information processing method, comprising: It includes the following steps: In the receiving step, image data obtained by the image capturing device is received, In the extraction step, a target object included in the image data is recognized, and only the image data in which the target object is included in an angle of view is extracted; In the excluding step, image data having inappropriate exposure and focus is excluded from the extracted images; In the identification step, the image data after the exclusion is identified; In the notification step, the user is notified of the labeling information after the identification. The target object is a piston ring or a ring land inside a cylinder of a marine diesel engine, In the excluding step, an image in which edge extraction for soot adhering to the piston ring or ring land cannot be performed is determined to be an image with an improper focus.

6. 6. The information processing method according to claim 5, Extracting image data in the extracting step; excluding image data in the excluding step; Identification of image data in the identifying step This is done based on pre-trained data.

7. 7. The information processing method according to claim 6, The information processing method further includes a learning step, The method is configured such that, in the learning step, the image data after classification is used as training data to generate or update the learned data.

8. In the information processing method according to any one of claims 5 to 7, The method, wherein the identifying step is configured to classify a state of the target object.

9. A program, A method for making a computer execute each step in the information processing system according to any one of claims 1 to 4.

Images