Determining anomalies on surfaces of or associated to industrial machines or components
Patent Information
- Application Number
- AE202602453
- Authority / Receiving Office
- AE · AE
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-01-22
- Filing Date
- 2025-01-17
Smart Images
Figure ABST_ABST
Abstract
Description
TITLEDetermining Anomalies on Surfaces of or associated to Industrial Machines or ComponentsDESCRIPTIONTECHNICAL FIELD
[0001] The subject matter disclosed herein relates to determining anomalies on surfaces of or associated to industrial machines or components of industrial machines.BACKGROUND ART
[0002] An “anomaly” in / of an industrial machine is a situation in / of the machine different from expected.
[0003] An “anomalous condition” may relate to a surface of an industrial machine or a surface of a component of an industrial machine, as well as to a surface associated to an industrial machine or a surface associated to a component of an industrial machine. For example, a surface associated to an industrial machine may be a horizontal surface of a ground floor where the machine rests. For example, a surface associated to a component of an industrial machine may be a vertical surface of a wall facing the component that is not necessarily a wall of the machine.
[0004] An oil stain on a surface of a ground floor where a machine rests is an “anomalous condition” if oil leakages from the machine are not expected and may be an indication that an “anomaly” is occurring or occurred in the machine (for example, an overpressure is occurring or a gasket broke down). It is to be noted that an oil stain on a surface associated to the machine may correspond to dripping of oil from a surface of the machine.
[0005] “Anomalous conditions” on surfaces may be detected by human inspection of these surfaces. Such solution to this problem is time-consuming as well as expensive, and requires skilled service personnel. For example, an oil stain on a ground floor may be not very easy to be detected especially if the floor is (naturally or voluntarily) washed before inspection and / or if a long time passes between stain occurrence and inspection and / or if the floor is made of a material that absorbs oil.
[0006] Similar considerations apply to other kinds of anomalous conditions of surfaces and / or to other surfaces (for example rust on a pipe) and / or surroundings of surfaces.SUMMARY
[0007] It would be desirable to determine “anomalous conditions” on surfaces and / or surroundings of surfaces in a way suitable for being used in industrial applications.
[0008] Such determination should be automatic, i.e. with no human intervention or with little human intervention, but still reliable.
[0009] One may think of capturing images of the surfaces to be analyzed and processing automatically such images in an electronic manner.
[0010] Traditional image processing of such images is complex and critical. First of all, a designer, e.g. an engineer, should define how the “anomalous condition” appears on a surface for example in terms of color and shape considering several possible perspectives and several possible illuminations. Afterwards, a corresponding electronic processing needs to be coded by a programmer, e.g. the same engineer or a different engineer. It is to be expected that such traditional image processing generates a high number of false positives and false negatives that need to be managed somehow, for example manually. Furthermore, each different “anomalous condition” requires distinct defining activity and distinct coding activity by engineers.
[0011] According to a first innovative aspect, the subject matter discloses herein relates to a computer-implemented method for determining an anomalous condition on a surface of an industrial machine or component or on a surface associated to an industrial machine or component. The method comprises the steps of: providing a set of sentence embedding features of a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on a surface; receiving an image of the surface and / or of surroundings of the surface; encoding said image thereby generating a set of image embedding features; determining a similarity value of a similarity score, wherein the similarity score corresponds to a level of similarity between said set of image embedding features and said set of sentence embedding features; comparing the determined similarity value with a predetermined similarity threshold; transmitting a result, the result being positive if the determined similarity value is higher than the predetermined similarity threshold.
[0012] According to a second innovative aspect, the subject matter discloses herein relates to a method for checking whether an industrial machine or component has an anomaly through a robotic vehicle configured to move and to repeatedly take pictures of portions of a surface and / or of surroundings of a surface through a camera and consequently to generate images. The method comprises the flowing steps for each of the generated images: carrying out the steps of the innovative determination method; signaling a result of the innovative determination method at least if the result is positive.
[0013] According to a third innovative aspect, the subject matter discloses herein relates to a system for checking whether an industrial machine or component has an anomaly. The system comprises: a robotic vehicle including a camera, wherein the robotic vehicle is configured to move and to repeatedly take pictures of portions of surface and / or of surroundings of a surface through the camera and consequently to generate images; an electronic processing unit coupled to the robotic vehicle and configured to carry out the steps of the innovative checking method.BRIEF DESCRIPTION OF THE DRAWINGS
[0014] A more complete appreciation of the disclosed embodiments of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:Fig. 1shows a block diagram of an exemplary industrial plant with several industrial machines resting on a ground floor surface,Fig. 2shows a block diagram of an embodiment of an innovative system,Fig. 3shows the block diagram of the industrial plant of Fig. 1 with portions of the ground floor surface along a first path of the innovative system of Fig. 2,Fig. 4shows the block diagram of the industrial plant of Fig. 1 with portions of the ground floor surface along a second path of the innovative system of Fig. 2,Fig. 5shows a flowchart of an embodiment of an innovative determination method,Fig. 6shows a flowchart of an embodiment of an innovative modelling method used in conjunction with the determination method of Fig. 5,Fig. 7shows plots of exemplary probability (e.g. density) functions generated thorough the modelling method of Fig. 6, andFig. 8shows a flowchart of an embodiment of an innovative checking method.DETAILED DESCRIPTION OF EMBODIMENTS
[0015] Determining in an automatic way an anomalous condition on a surface of / associated to an industrial machine is a difficult task if the result of the determination should be reliable. The task is difficult even if the anomalous condition is predetermined, for example the presence of a stain of machine oil on a surface of a ground floor where the industrial machine rests.
[0016] According to the subject matter disclosed herein, instead of processing a captured image with the objective of identifying in the image the anomaly, for example the above-mentioned oil stain, a comparison is made between the captured image (once suitably encoded) and a sentence (once suitably encoded). The sentence is a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on a surface, for example “the image shows a ground floor with at least one oil stain thereon”. The same kind of encoding is applied to both the sentence and the image; the objective is to obtain a so-called (in the field of Artificial Intelligence – see for example the publication of Vicki Boykis entitled “What are embeddings” at “https: / / vickiboykis.com / what_are_embeddings / next.html”) “embedding” or, more precisely, a “set of embedding features”, typically a vector of real numbers, for both the sentence and the image. The comparison may be made between the two vectors, which is a relatively easy task. If the two vectors have the same number of components, the “comparison” may be carried out through the so-called “scalar product” between the two vectors. The result of the “comparison” is a number that indicates the similarity between the two vectors: the higher is the number the more similar are the two vectors. If the two compared vectors are (numerically) similar, the sentence corresponding to the first vector describes correctly the image corresponding to the second vector, in other words the image shows the anomaly.
[0017] A technique for comparing sentences and images (through embeddings) is known, as such, for example from the article by Alec Radford et al. entitled “Learning Transferable Visual Models From Natural Language Supervision” in the Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021; corresponding code and pre-trained model weights have been released at https: / / github.com / OpenAI / CLIP under the name “CLIP”. CLIP, especially if already trained, may be used for implementing innovative methods and systems according to the subject matter disclosed herein. As explained in this article, learning image representations from text was known and possible even before this article (see e.g. architectures named VirTex and ConVIRT). According to this article (see e.g. Figure 1), CLIP is able to automatically associated an image to a text message chosen in a set of predetermined text messages.
[0018] Fig. 1 shows an exemplary industrial plant 1000 where the innovative methods and the innovative systems may be applied and used. The plant 1000 includes several industrial machines, for example six industrial machines 100, 200, 300, 400, 500 and 600. These industrial machines rest for example (indirectly) on a same ground floor 10 through a corresponding base not shown in the figures. For the sake of simplicity, the industrial machines 100, 200, 300, 400, 500 and 600 are shown in Fig. 1 as having the same shape and as positioned in a regular manner; however, this is not to be construed as a limitation; in fact, these industrial machines may indeed be different from each other and / or positioned in an irregular manner.
[0019] Fig. 2 shows inter alia an exemplary robotic vehicle 2200 that may be used for the innovative methods and for the innovative systems for example in the industrial plant 1000 of Fig. 1. The robotic vehicle 2200 is configured to move (see e.g. wheels 2240) and to take pictures (see e.g. camera 2230). According to the embodiment of Fig. 2, the robotic vehicle 2200 is configured to move (preferably automatically) on the ground floor 10 and to take pictures (preferably automatically) of portions of the surface of the ground floor 10.
[0020] Fig. 3 shows the industrial plant 1000 of Fig. 1 and a first path 21 of the robotic vehicle 2200 on the ground floor 10. The robotic vehicle 2200 takes pictures of portions of the surface of the ground floor 10 along path 21; these portions are for example circles as shown in Fig. 3 (alternatively, the shape may be different). According to this embodiment, the portions do not overlap (alternatively, they overlap). It is possible that the robotic vehicle 2200 stops before taking a picture and then restarts after taking a picture; alternatively, the robotic vehicle 220 takes pictures while moving, possibly slowly moving.
[0021] Fig. 4 shows the industrial plant 1000 of Fig. 1 and a second path 22 of the robotic vehicle 2200 on the ground floor 10. Similar considerations apply to both path 21 and path 22, even if they are different.
[0022] According to a first possibility, the robotic vehicle 2200 follows always the first path 21. According to a second possibility, the robotic vehicle 2200 follows always the second path 22. According to a third possibility, the robotic vehicle 2200 follows both the first path 21 and the second path 22. It is clear that other paths are possible and other possibilities are possible. Ideally, the whole surface of the ground floor 10 should be photographed.
[0023] Each of the pictures taken by the robotic vehicle 2200, once transformed into an electronic image, i.e. an “image”, should be analyzed in order to determine an anomalous condition of the surface under consideration. According to the embodiment of figures 1, 3 and 4, it is a surface associated to a plurality of industrial machines, specifically a surface of a ground floor that supports (indirectly, i.e. through bases) several industrial machines.
[0024] Analysis of an image coming from e.g. the robotic vehicle serves for determining an anomalous condition on the surface of e.g. the ground floor, in general a surface of an industrial machine or component or a surface associated to an industrial machine or component.
[0025] As shown in the exemplary flowchart 5000 of Fig. 5, such innovative (determination) method is implemented by a computer system and comprises in general the steps of:b) providing (see e.g. step 5100) a set of sentence embedding features of a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on the surface under consideration (such sentence may be for example “the image shows a ground floor with at least one oil stain thereon”),c) receiving (see e.g. box 5200) an image,d) encoding (see e.g. box 5300) said image thereby generating a set of image embedding features,e) determining (see e.g. box 5400) a similarity value of a similarity score, wherein the similarity score corresponds to a level of similarity between said set of image embedding features and said set of sentence embedding features,f) comparing (see e.g. box 5500) the determined similarity value with a predetermined similarity threshold, andg) transmitting (see e.g. box 5600) a result, the result being positive if the determined similarity value is higher than the predetermined similarity threshold.
[0026] The image referred to at step “c” may be an image of the surface under consideration (i.e. a surface of an industrial machine or of a component of an industrial machine or a surface associated to an industrial machine or to a component of an industrial machine) or of surroundings of the surface under consideration; specifically, the image may show a portion of the surface, or a portion of the surroundings of the surface, or both a portion of the surface and a portion of the surroundings. For example, if the anomaly of interest is oil dripping or vapor / gas ejecting from a junction, the image to be processed may show a portion of a surface of the machine where the junction is located, or a surrounding of this surface portion (i. e. a surroundings of the junction), or both the surface portion and the surroundings; depending on the case and the application, the position and size of the surroundings may vary.
[0027] These steps may be repeated for several images, for example for each image generated by the robotic vehicle starting from each picture taken, that is, considering Fig. 3, for each portion of the surface of ground floor 10 along path 21, and, considering Fig. 4, for each portion of the surface of ground floor 10 along path 22.
[0028] It is to be noted that the innovative method as set out in general above does not specify 1) how and when the sentence embedding features are determined and 2) how the similarity value of the similarity score is determined and 3) how and when the similarity threshold is determined. Regarding the sentence embedding features, their determination may be part of activities preceding step “b” or included in step “b”. Regarding the similarity threshold, its determination may be part of activities preceding step “b” and included for example in a step “a” (shown in Fig. 6 according to an embodiment thereof) that may be common to the analysis of several distinct images and therefore not shown in the flowchart of Fig. 5.
[0029] According to advantageous embodiments of the innovative determination method, the anomalous condition corresponds to the presence of a stain of machine working fluid on a machine associated surface, the machine working fluid being in particular oil, and the machine associated surface corresponds to a surface that supports directly or indirectly the industrial machine.
[0030] According to other advantageous embodiments of the innovative determination method, the anomalous condition corresponds to the presence of a rust formation on a machine surface and the machine surface corresponds to a surface of a pipe of the industrial machine.
[0031] According to still other advantageous embodiments of the innovative determination method, the anomalous condition corresponds to liquid dripping or vapor / gas ejecting from a machine or a component of a machine, and the surface to be observed may correspond to a surface of the machine where the liquid or vapor or gas exits from or to a surface of the component where the liquid or vapor or gas exits from or to surroundings of such surfaces that may show the liquid dripping or the vapor / gas ejecting.
[0032] The sentence embedding features mentioned at step “b” are necessary for performing the determination of the similarity value of the similarity score at step “e”. According to some embodiments of the innovative determination method, a set of them is determined somehow by a first computer system and is stored in a digital memory of a second computer system, i.e. the computer system carrying out the innovative determination method. The first and second computer systems may be embodied as distinct computer systems or as a single computer system. In the case of distinct systems, step “b” may comprise the step of receiving (at some point in time) the set of sentence embedding features. In the case of a single system, step “b” may comprise the steps of receiving (at some point in time) the natural language sentence expressing the concept relating to a presence of a predetermined anomaly on a surface, and encoding said natural language sentence thereby generating the set of sentence embedding features.
[0033] The determination of the similarity value of the similarity score at step “e” (starting from the set of image embedding features and the set of sentence embedding features) may be done in many different ways. A very effective and simple way is to calculate a so-called “scalar product” between the set of image embedding features and the set of sentence embedding features; this is possible if the two sets have the same number of components; if the two sets do not have the same number of components, it is possible to transform / adapt any or each of the two vectors through e.g. a corresponding neural network (e.g. fully-connected) layer having the function, for example, of reducing the number of variables (i.e. components of the vector). Other ways for determining the similarity, in particular as distances, are described in general for example in the article of Mahmoud Harmouch entitled “17 types of similarity and dissimilarity measures used in data science” published by “Towards Data Science” in 2021.
[0034] Typically, the above-mentioned steps “b”, “c”, “d”, “e”, “f”, “g” are carried out during an operating phase, while the predetermined similarity threshold is calculated prior to step “b” during a training phase preceding the operating phase.
[0035] The operating phase is a phase aimed at determining an anomalous condition, if present. For example, considering Fig. 3 or Fig. 4, during an operating phase the robotic vehicle 2200 of Fig. 2 moves along the path 21 or 22, repeatedly takes pictures of portions of the surface 10 through the camera 2230, consequently generates images, and the images are processed (at least through steps d, e and f) for example by an electronic unit 2420 of a control station 2400 coupled to the robotic vehicle 2200 (the combination of the robotic vehicle 2200 and the control station 2400 is an innovative checking system 2000). It is to be noted that, considering Fig. 3 or Fig. 4, a single operating phase may allow to check several industrial machines or components of several industrial machines.
[0036] The training phase precedes the operating phase. For example, the training phase of the innovative checking system 2000 of Fig. 2 may be carried out when designing the system and / or when manufacturing the system and / or when installing the system; an advantageous possibility consists in training the system partially at the manufacturing site and partially at the installing site. It is to be noted that while the word “training” is used herein with reference to the determination of the “optimal” (or at least “good”) similarity threshold, according to a different terminology choice, this word might refer to the image-sentence comparison; in this case, the determination of the “optimal” similarity threshold may be considered part of a “modelling” phase.
[0037] An important aspect of the innovative method is the determination of the similarity threshold that may be considered step “a” of an innovative method (see dashed portion of the flow chart 5000 in Fig. 5). An embodiment of step “a”, preceding step “b”, is described in the following with the help of the flow chart 6000 of Fig. 6 and the plots 7000 in Fig. 7.
[0038] The predetermined similarity threshold may be calculated through the steps (more correctly, “sub-steps”) of:a1) receiving (see e.g. box 6100) a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on a surface (such sentence may be for example “the image shows a ground floor with at least one oil stain thereon” and is the same sentence used at step “b” of the innovative determination method),a2) encoding (see e.g. box 6100) said natural language sentence thereby generating a set of sentence embedding features,a3) receiving (see e.g. box 6200) a first plurality of images showing a surface or surroundings with said predetermined anomaly thereon,a4) encoding (see e.g. box 6200) each of the images of said first plurality thereby generating a corresponding first plurality of sets of image embedding features,a5) determining (see e.g. box 6200) a first plurality of values of a similarity score, wherein the similarity score corresponds to a level of similarity between said sets of image embedding features of said first plurality and said set of sentence embedding features,a6) generating (see e.g. box 6200) a first probability function (see e.g. function PF1) based on said first plurality of similarity score values,a7) receiving (see e.g. box 6300) a second plurality of images showing a surface or surroundings without said predetermined anomaly thereon,a8) encoding (see e.g. box 6300) each of the images of said second plurality thereby generating a corresponding second plurality of sets of image embedding features,a9) determining (see e.g. box 6300) a second plurality of values of a similarity score, wherein the similarity score corresponds to a level of similarity between said sets of image embedding features of said second plurality and said set of sentence embedding features,a10) generating (see e.g. box 6300) a second probability function (see e.g. function PF2) based on said second plurality of similarity score values,a11) finding (see e.g. box 6400) an intersection between said first and second probability functions (see e.g. functions PF1 and PF2),a12) determining (see e.g. box 6500) a value of similarity score of said intersection,anda13) equating (see e.g. box 6600) the predetermined threshold (see e.g. value ST) to said determined value of similarity score plus possibly a value of similarity margin.
[0039] It is to be noted that the determination of the similarity threshold includes some processing analogous to some processing of the innovative determination method. In other words, some processing during the training phase is analogous to some processing in the operating phase. Preferably, analogous processing could / should be made exactly in the same way. For example, the sentence mentioned at steps “a1” could / should be identical to the sentence mentioned at step “b”. For example, the determination of the values of the similarity scores at steps “a5” and “a9” could / should preferably be made exactly as the determination of the value of the similarity score at step “e”, preferably through a so-called “scalar product” between a set of image embedding features and a set of sentence embedding features.
[0040] Additionally, it is to be noted that the probability functions (see e.g. functions PF1 and PF2) are generated starting from training images that are “annotated” only by dividing them into two pluralities: a first plurality of “anomalous images”, i.e. showing a sample of the anomaly of interest (for example at least one “oil stain on a ground floor”) and a second plurality of “non-anomalous images”, i.e. not showing any sample of the anomaly of interest (for example no “oil stain on a ground floor”). No further “annotations” to the training images are necessary (for example, it is not necessary to mark the contour of the stain in the images).
[0041] Finally, it is to be noted that the first probability function and the second probability function are generated starting from the same sentence.
[0042] While Fig. 7 shows for the sake of simplicity functions PF1 and PF2 as identical to gaussian curves, in practice they are only (more or less) similar to gaussian curves. The intersection of the two curves identifies a value ST of a similarity score that is used as a “Similarity Threshold”. In fact, if a computed similarity value of the similarity score of an image is above the value ST, one may infer that this image shows an anomaly, while, if a computed similarity value of the similarity score of an image is below the value ST, one may infer that this image does not show an anomaly. However, differently from what is shown in Fig. 7, it may be advantageous to add to the similarity value of the intersection a similarity margin so that the similarity threshold ST is different from (typically higher than) the similarity value of the intersection.
[0043] Functions PF1 and PF2 are “probability functions”, more precisely, so-called (in the field of statistics) “probability density functions” as they correspond to the probabilities of random variables being any “real number” in the range between e.g. 0 and e.g. 1. Alternatively, such functions may be so-called (in the field of statistics) “probability mass functions” wherein the random variables may assume discrete values in the range between e.g. 0 and e.g. 10 or 100.
[0044] Innovative determination methods have just been described able to determine whether an image shows an “anomalous” condition. Results of such methods, that are for example positive in case of “anomalous” conditions, may be transmitted in the sense of provided to an entity external to the one in charge of such determination. It may be for example a distinct computer system or a distinct processor of the same computer system or a distinct processing task of the same processor of the same computer system; so transmission can be local or remote.
[0045] Advantageously and typically, such innovative determination methods (and their results) may be used in methods for checking whether an industrial machine (or component thereof) has an anomaly. Such checking methods are also innovative if they are based on innovative determination methods.
[0046] In general, such innovative checking methods make use of a robotic vehicle (such as for example vehicle 2200 in Fig. 2) configured to move and to repeatedly take pictures of portions of a surface of interest (and / or of surroundings of such portions) through a camera (such as for example camera 2230 in Fig. 2) and consequently to generate images. The movement of the robotic vehicle is advantageously automatic, for example under the control of a control unit (such as for example unit 2220 in Fig. 2) and / or of a control station (such as for example station 2400 in Fig. 2). The taking of pictures by the robotic vehicle is advantageously automatic for example under the control of a control unit (such as for example unit 2220 in Fig. 2) and / or of a control station (such as for example station 2400 in Fig. 2). However, human intervention is not to be excluded especially in unexpected situations.
[0047] In general, an innovative checking method comprises the flowing steps for each of the generated images (see for example flowchart 8000 of the embodiment of Fig. 8):m) carrying out (see e.g. box 8100) the steps of an innovative determination method (see e.g. box 5000), andn) signaling (see e.g. box 8200) a result of the innovative determination method (see e.g. box 5000) at least if the result is positive.
[0048] As shown by the loop configuration in Fig. 8, steps “m” and “n” are typically repeated for each new image.
[0049] The robotic vehicle for an innovative checking method may be an unmanned terrestrial vehicle or an unmanned aerial vehicle or an unmanned underwater vehicle.
[0050] According to advantageous embodiments (see for example figures 1-4), the robotic vehicle is an unmanned terrestrial vehicle, the robotic vehicle is configured to move (e.g. automatically) on a surface supporting (directly or indirectly) an industrial machine, the robotic vehicle is configured to take pictures (e.g. automatically) of portions of the surface supporting (directly or indirectly) the industrial machine.
[0051] According to advantageous embodiments (see for example figures 3-4), the robotic vehicle is configured to move automatically along at least one predetermined path. It is to be noted, referring for example to the paths 22 and 22 in Fig. 3 and Fig. 4, that a single path allows to check several machines or components of several machines; in this case, a single innovative checking method regards several machines or components.
[0052] According to advantageous embodiments, the robotic vehicle is configured to take pictures automatically aiming in at least one predetermined and fixed direction (for example, in the embodiment of Fig. 2, the camera 2230 may aim always downward towards the ground floor) or in a plurality of predetermined and fixed directions. The word “pictures” is to be construed as including also “panoramic shooting” captured through a picture camera or a video camera. The possibility of “active sensing” will be described afterwards.
[0053] According to some embodiments of the innovative checking method, if a result of a determination is positive or “close to positive” (in other words an anomalous condition seems to be present), it may be provided another step “o” after step “n”. In particular, according to step “o”, taking at least one further picture of a same portion of the surface is taken by the robotic vehicle; the further picture may be taken from the same position or from a (slightly) different position; the further picture may be taken aiming in the same direction or in a (slightly) different direction. This may increase reliability of the checking action. The expression “close to positive” may mean that the computed similarity value of the similarity score for a certain image is not greater than the predetermined similarity threshold, but is close; for example, the threshold is 0.65 and the computed value is 0.62. Alternatively, the expression “close to positive” may mean that the computed similarity value is in the range between the similarity threshold and the sum of the similarity threshold and the similarity margin.
[0054] The functions PF1 and PF2 in Fig. 7 correspond to curves being quite wide and quite overlapping. It would be advantageous that such curves be less wide and less overlapping; this would mean a lower likelihood of error in the determination of “anomalous condition” during operation. A way to achieve such goal is to fine tune, i.e. to optimize, the natural language sentence, in particular identifying a very good or the best sentence able to maximize the model performances (in particular, the accuracy of the determination at step “e”), i.e. distinguishing images corresponding to the desired concept and images not corresponding to it. Also in this case, the fine-tuned natural language sentence used for determining the similarity value of the similarity score is used both during training as well as during operation. Another possibility would be to use concurrently multiple sentences expressing the same concept with different words and to consider multiple comparison results.
[0055] As already mentioned, innovative checking methods make use of a robotic vehicle. Fig. 2 shows an embodiment of an innovative system 2000 including such robotic vehicle for checking whether an industrial machine or component has an anomaly, i.e. an “innovative checking system”.
[0056] The system 2000 comprises a robotic vehicle 2200 and a control station 2400. The robotic vehicle 2200 is configured to move for example on the ground floor 10 of the industrial plant 1000. Both the vehicle 2200 and the station 2400 includes a control unit, respectively 2220 and 2420, positioned for example inside a casing, respectively 2210 and 2410; the combination of these two units may be considered an “electronic processing unit” of the system 2000 that is distributed between two distinct components of the system usually positioned remote from each other (in fact the station 2400 is fixed while the vehicle 2200 is moving). According to the embodiment of Fig. 2, the vehicle 2200 includes a picture camera 2230 (alternatively, a video camera) for taking pictures, in particular for repeatedly taking pictures of portions of the surface of ground floor 10, and wheels 2240 (alternatively, other motion means such as legs, wings and propellers) for moving, in particular for moving on the ground floor 10.
[0057] According to the embodiment of Fig. 2, the robotic vehicle 2200 and the control station 2400 are configured to communicate wirelessly. In Fig. 2, an antenna 2250 of the vehicle 2200 (connected to the control unit 2220) and an antenna 2450 of the station 2400 (connected to the control unit 2420) are shown by way of example used for such wireless communication; the antennas are respectively connected to the control units. The station 2400 may transmit commands to the vehicle 2200 and the vehicle 220 may transmit data, for example images (that may be pre-processed or not), to the station 2400; the station 2400 may transmit also data to the vehicle 2200, for example configuration data and / or software.
[0058] The moving of the vehicle 2200 and the taking pictures of the vehicle 2200, in particular by the camera 2230, are under control of the “electronic processing unit” of the system 2000. Depending on the embodiment, some actions are under direct control of the (local) control unit 2220 of the vehicle (that is connected to the camera 2230 and to the wheels 2240) and some actions are under indirect control of the (remote) control unit 2420 of the station. The moving and the taking of pictures are advantageously automatic in the sense that no human intervention is necessary for performing them. However, human intervention may not be excluded especially in unexpected situations.
[0059] An innovative checking method is carried out thanks to the “electronic processing unit” of the system 2000; some actions may be performed thanks to the control unit 2220 of the vehicle and some actions may be performed thanks to the control unit 2420 of the station.
[0060] As it is apparent for the person skilled in the art, an innovative checking system may have several functionalities in addition to those disclosed above. For example, the picture camera 2230 in combination with the control unit 2220 may implement a first functionality so-called “panoramic shooting” and / or a second functionality so-called “active sensing”. According to the first functionality, the camera takes a first picture aiming in a first direction, then rotates a bit, than takes a second picture aiming in a second direction, … and finally all the taken pictures are assembled by the control unit to form a single picture. According to the second functionality, the camera takes a picture, the picture is analyzed by the control unit and, if the quality of the picture is considered insufficient, another picture is taken possibly by activating a lighting device and / or by (slightly) moving (for example rotating) the camera.
[0061] The subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structural means disclosed in this specification and structural equivalents thereof, or in combinations of them. The subject matter described herein can be implemented as one or more computer program products, such as one or more computer programs tangibly embodied in an information carrier (e.g., in a machine readable storage device), or embodied in a propagated signal, for execution by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). A computer program (also known as a program, software, software application, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file. A program can be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.
[0062] The processes and logic flows described in this specification, including the method steps of the subject matter described herein, can be performed by one or more programmable processors executing one or more computer programs to perform functions of the subject matter described herein by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus of the subject matter described herein can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
[0063] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processor of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks, (e.g., internal hard disks or removable disks); magneto optical disks; and optical disks (e.g., CD and DVD disks). The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0064] To provide for interaction with a user, the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, (e.g., a mouse or a trackball), by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.
[0065] The techniques described herein can be implemented using one or more modules. As used herein, the term “module” refers to computing software, firmware, hardware, and / or various combinations thereof. At a minimum, however, modules are not to be interpreted as software that is not implemented on hardware, firmware, or recorded on a non-transitory processor readable recordable storage medium (i.e., modules are not software per se). Indeed “module” is to be interpreted to always include at least some physical, non-transitory hardware such as a part of a processor or computer. Two different modules can share the same physical hardware (e.g., two different modules can use the same processor and network interface). The modules described herein can be combined, integrated, separated, and / or duplicated to support various applications. Also, a function described herein as being performed at a particular module can be performed at one or more other modules and / or by one or more other devices instead of or in addition to the function performed at the particular module. Further, the modules can be implemented across multiple devices and / or other components local or remote to one another. Additionally, the modules can be moved from one device and added to another device, and / or can be included in both devices.
[0066] The subject matter described herein can be implemented in a computing system that includes a back end component (e.g., a data server), a middleware component (e.g., an application server), or a front end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein), or any combination of such back end, middleware, and front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
[0067] Although some specific embodiments have been described herein, other embodiments are within the scope and spirit of the subject matter as per the annexed claims.
[0068] It is noted that one or more references are incorporated herein. To the extent that any of the incorporated material is inconsistent with the present disclosure, the present disclosure shall control. Furthermore, to the extent necessary, material incorporated by reference herein should be disregarded if necessary to preserve the validity of the claims.
Claims
1. A computer-implemented method (5000) for determining an anomalous condition on a surface of an industrial machine (100) or component or on a surface (10) associated to an industrial machine (100) or component,wherein the method comprises the steps of:a) calculating a predetermined similarity threshold, b) providing (5100) a set of sentence embedding features of a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on a surface, c) receiving (5200) an image of the surface (10) and / or of surroundings of the surface, d) encoding (5300) said image thereby generating a set of image embedding features, e) determining (5400) a similarity value of a similarity score, wherein the similarity score corresponds to a level of similarity between said set of image embedding features and said set of sentence embedding features, f) comparing (5500) the determined similarity value with the predetermined similarity threshold, and g) transmitting (5600) a result, the result being positive if the determined similarity value is higher than the predetermined similarity threshold.
2. The method of claim 1, wherein said predetermined similarity threshold is calculated through the steps of: a1) receiving (6100) a natural language sentence expressing a concept relating to a presence of a predetermined anomaly on a surface, a2) encoding (6100) said natural language sentence thereby generating a set of sentence embedding features, a3) receiving (6200) a first plurality of images showing a surface or surroundings with said predetermined anomaly thereon, a4) encoding (6200) each of the images of said first plurality thereby generating a corresponding first plurality of sets of image embedding features, a5) determining (6200) a first plurality of values of a similarity score, wherein the similarity score corresponds to a level of similarity between said sets of image embedding features of said first plurality and said set of sentence embedding features, a6) generating (6200) a first probability function (PF1) based on said first plurality of similarity score values, a7) receiving (6300) a second plurality of images showing a surface or surroundings without said predetermined anomaly thereon, a8) encoding (6300) each of the images of said second plurality thereby generating a corresponding second plurality of sets of image embedding features, a9) determining (6300) a second plurality of values of a similarity score, wherein the similarity score corresponds to a level of similarity between said sets of image embedding features of said second plurality and said set of sentence embedding features, a10) generating (6300) a second probability function (PF2) based on said second plurality of similarity score values, a11) finding (6400) an intersection between said first and second probability functions, a12) determining (6500) a value of similarity score of said intersection,and a13) equating (6600) the predetermined threshold to said determined value of similarity score plus possibly a value of similarity margin.
3. The method of claim 2,wherein each of steps “a5” and “a9” comprises the step of:calculating a scalar product between a set of image embedding features and a set of sentence embedding features.
4. The method of claim 1,wherein the anomalous condition corresponds to the presence of a stain of machine working fluid on a machine associated surface, the machine working fluid being in particular oil,wherein the machine associated surface corresponds to a surface that supports the industrial machine.
5. The method of claim 1,wherein step “b” comprises the step of: b1) storing the set of sentence embedding features.
6. The method of claim 1,wherein step “b” comprises the step of: b2) receiving the set of sentence embedding features.
7. The method of claim 1,wherein step “b” the comprises the steps of: b31) receiving the natural language sentence expressing the concept relating to a presence of a predetermined anomaly on a surface, b32) encoding said natural language sentence thereby generating the set of sentence embedding features.
8. The method of claim 1,wherein step “e” comprises the step of: e1) calculating a scalar product between said set of image embedding features and said set of sentence embedding features.
9. The method of claim 1,wherein the natural language sentence or the set of sentence embedding features derives or derive from a preliminary optimization process aimed at maximizing accuracy of the determination at step “e”.
10. The method of claim 1 and 2,wherein steps “b”, “c”, “d”, “e”, “f”, “g” are carried out during an operating phase, andwherein said predetermined similarity threshold is calculated prior to step “b” during a training phase preceding said operating phase.
11. A method (8000) for checking whether an industrial machine (100) or component has an anomaly through a robotic vehicle (2200) configured to move and to repeatedly take pictures of portions of a surface (10) and / or of surroundings of a surface through a camera (2230) and consequently to generate images, wherein the method comprises the flowing steps for each of the generated images: m) carrying out (8100) the steps of the determination method (5000) of claim 1, and n) signaling (8200) a result of the determination method (5000) of claim 1 at least if the result is positive.
12. The method of claim 11,wherein the robotic vehicle is an unmanned terrestrial vehicle,wherein the robotic vehicle is configured to move on a surface supporting the industrial machine, andwherein the robotic vehicle is configured to take pictures of portions of the surface supporting the industrial machine.
13. The method of claim 11,wherein the robotic vehicle is configured to move automatically along a predetermined path.
14. The method of claim 11,wherein the robotic vehicle is configured to take pictures automatically aiming in at least one predetermined and fixed direction.
15. The method of claim 14,wherein the robotic vehicle is configured to take pictures automatically aiming in a plurality of predetermined and fixed directions.
16. The method of claim 11,wherein the method comprises the flowing step after step “n” if the result is positive or close to positive: o) taking at least one further picture of a same portion of the surface.
17. A system (2000) for checking whether an industrial machine (100) or component has an anomaly, wherein the system (2000) comprises:a robotic vehicle (2200) including a camera (2230), wherein the robotic vehicle (2200) is configured to move and to repeatedly take pictures of portions of surface (10) and / or of surroundings of a surface through the camera (2230) and consequently to generate images, andan electronic processing unit (2220, 2420) coupled to the camera (2230), wherein the electronic processing unit (2220, 2420) is configured to carry out the steps of the checking method (8000) of claim 11.