Method and system for controlling the speed of an object moving device

Abstract

An embodiment of this method for controlling the speed of an object moving device (OBJi) comprises the following steps: - determining (E10) a direction (DIROP) of an operator's gaze; - detecting (E20), in a video stream (FV) at least one image (IMGOBJi) of one of said moving objects; - determining (E30) a position of said object (OBJi); - determining (E45) an interaction duration (DI) during which the operator (OP) looks at said object (OBJi) from the position (POSOi) of said object (OBJi) and the direction (DIROP) of the operator's gaze; - controlling (E50) a speed of movement of said object moving device (TR) (OBJi) as a function of said duration (DI). Fig. 5

Description

Title of the invention: Method and system for controlling the speed of an object moving device Prior art

[0001] The present disclosure is situated in the general context of quality control of objects, for example at the output of a production line, for example in an industrial environment.

[0002] In industry, it is known that part verification and quality control tasks can be hampered by machine throughput, leading to verification errors and the production of non-conforming parts.

[0003] The present disclosure aims at a solution to improve this situation. Subject matter and summary of the invention

[0004] More specifically, the present disclosure relates to a method for controlling the speed of a device for moving objects as a function of at least one occurrence of interaction of an operator with at least one portion to be inspected of at least one of said objects.

[0005] Correlatively, the present disclosure relates to a system for controlling the speed of a device for moving objects as a function of at least one occurrence of interaction of an operator with at least one portion to be inspected of at least one of said objects.

[0006] The object moving device may be of any nature. It may, for example, consist of a conveyor belt.

[0007] In the current state of the art, due to the large amount of information provided by the devices in a production line and the often constant speed of the conveyor belts, it cannot be ensured that the operator has had the necessary time to examine all the parts.

[0008] The method and system of the present disclosure propose to adapt the speed of presentation of the objects to the user to ensure that the operator has been able to interact with the objects in order to be able to inspect or examine them.

[0009] In one embodiment, the occurrence of an interaction of the operator with at least one portion of the object is determined by measuring a duration of interaction of the operator with this portion of the object.

[0010] Thus, when scrolling through the objects, by measuring the duration of interaction of the operator with the objects or portions of these objects, the method makes it possible to check whether the operator has had time to check the conformity of each object. As soon as an object has not had time to be checked, the method can slow down the belt or even stop it so that the operator can complete his task.

[0011] In one embodiment, the interaction duration is a duration of observation of said at least one object by the operator. In a particular embodiment, the method comprises the following steps:

[0012] - determination of a direction of the operator's gaze;

[0013] - detection, in a video stream, of at least one image of one of said objects in investment;

[0014] - determination of a position of said object;

[0015] - determination of an interaction duration during which the operator looks at said object from the position of said object and the direction of the operator's gaze;

[0016] - control of a speed of movement of said object moving device in function of the said duration.

[0017] Thus, the present disclosure proposes to use an eye-tracking method to measure the duration with which an operator observes an object or a portion of an object and to adapt the speed of presentation of the next objects to this operator as a function of this duration.

[0018] In this embodiment, the present disclosure more specifically proposes to track the eye movements of the operator to evaluate the objects or portions of objects that he has examined. By using eye tracking, the system ensures that the activity has been completely carried out, adjusting the speed of the machine accordingly. This method makes it possible to detect fatigue or the need to slow down the pace, taking into account the needs of the operator by tracking his gaze.

[0019] In one embodiment, the method comprises a step of identifying a digital twin of the object detected in the video stream, the interaction duration being a duration during which the direction of the operator's gaze crosses a volume encompassing the digital twin of this object.

[0020] This embodiment makes it possible in particular to transfer the calculation of the interaction duration to a dedicated machine and process. In particular, this determination can be made at the time of measuring the tracking of the direction of gaze in the headset, the latter knowing the digital twin of the part. It can also be transferred to a server which has knowledge of both the direction of gaze and the twin of the object.

[0021] This embodiment makes it possible, by working on simple volumes, to greatly reduce the complexity of the algorithm making it possible to determine whether the operator is actually looking at an object.

[0022] The embodiment of the present disclosure based on eye tracking, a technique for recording eye movements, to determine the direction of the operator's gaze is very precise and efficient.

[0023] It uses precise cameras and powerful algorithms to accurately measure eye position and movement.

[0024] In another embodiment, the interaction duration is a duration of manipulation of the object by the operator. For this purpose, simpler visual means can be used to detect the presence of hands on the object, or even mechanical means to detect that the object has been lifted, by measuring the weight on the mat for example.

[0025] In one embodiment, the control method, the subject of this disclosure, comprises an analysis of the images of the video stream to determine whether or not an object detected in this stream has a defect.

[0026] This analysis can for example be implemented by an artificial intelligence algorithm trained from images of objects presenting defects and images of objects not presenting defects.

[0027] The integration of artificial intelligence into factories introduces a margin of error, requiring human validation to confirm or reject results.

[0028] The present disclosure proposes to carry out this validation by human inspection of the objects, the validation rate being adjusted according to the operator's ability to look at an object for a sufficiently long time, this ability being likely to vary over time, in particular according to his state of concentration or fatigue.

[0029] In one embodiment, the method comprises a step of projecting, onto the screen of a virtual reality headset of the operator, an image obtained from said image detected in the video stream.

[0030] This mechanism advantageously makes it possible to encourage the user to examine a particular part by making it appear, for example, highlighted on the screen.

[0031] In one embodiment, the image projected onto the screen of the operator's virtual reality headset depends on the result of the fault analysis.

[0032] Advantageously, only parts presenting a defect, or on the contrary only parts assumed to be perfect can be projected onto the operator's helmet.

[0033] In another embodiment, the projected pieces are randomly selected assumed perfect pieces.

[0034] Alternatively, depending on whether a defect is detected or not, the images of the objects can be projected differently, for example in different colors.

[0035] The present disclosure also relates to a computer program comprising instructions for executing the steps of the method for controlling the speed of a device for moving objects when said program is executed by a computer.

[0036] This program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code. object, such as in a partially compiled form, or in any other desirable form.

[0037] The present disclosure also relates to a computer-readable information medium, and comprising instructions of a computer program as mentioned above. The information medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM, a non-volatile memory of the flash type or even a magnetic recording means, for example a hard disk. Furthermore, the information medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The program according to the present disclosure may in particular be downloaded from a network of the Internet type.Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question. Brief description of the drawings

[0038] Other characteristics and advantages of the present disclosure will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof without any limiting character. In the figures:

[0039] [Fig-1] [Fig.l] represents an operator in front of an example of a device for de placement of objects;

[0040] [Fig.2] [Fig.2] represents a speed control system of a device of moving objects according to one embodiment of the present disclosure;

[0041] [Fig.3] [Fig.3] represents a server that can be used in a system of control in accordance with another embodiment of the present disclosure;

[0042] [Fig.4] [Fig.4] represents a server that can be used in a system of control in accordance with another embodiment of the present disclosure;

[0043] [Fig.5] [Fig.5] represents the hardware architecture of a server that can be used in a control system according to an embodiment of the present disclosure; and

[0044] [Fig.6] [Fig.6] represents in the form of an organizational chart the main stages of a Method for controlling the speed of an object moving device according to a particular embodiment of the present disclosure Description of the embodiments

[0045] [Fig.l] represents an operator OP in front of a conveyor belt TR configured to drive objects OBJ;.

[0046] In the embodiment described herein, it is assumed that the treadmill is driven by two rollers mounted on a fixed frame and that the operator is static relative to a REF reference.

[0047] The objects OBJ; therefore move in front of the operator OP and the conveyor belt TR constitutes a device for moving objects within the meaning of the present disclosure.

[0048] In this example, the operator OP is equipped with a virtual reality headset CRV.

[0049] In the embodiment described herein, the virtual reality headset CRV is provided of a CAMc camera and MTC processing means (processor, memories, computer program) configured to determine: - the POSCrv position (and orientations) of the CRV helmet; and - the direction DIR0P of the operator OP's gaze in the REF reference frame.

[0050] To simplify the description, below, the POSCrv position is here a data structure which includes both the coordinates of the CRV helmet and its orientation according to the three orthonormal dimensions of the REF reference frame.

[0051] It will be assumed in the embodiment described here that the direction DIR0P of the gaze of the operator OP is obtained by determining an average direction calculated from the directions of the gaze of each of the eyes of the operator OP.

[0052] [Fig.2] represents a control system SYS according to a particular embodiment.

[0053] In the exemplary embodiment described here, the control system SYS comprises: - the CRV virtual reality headset, - the TR treadmill (object moving device); - a CAM camera; and - an SRVi server, these elements being interconnected between them by an unreferenced network.

[0054] In the embodiment of [Fig.2], the virtual reality headset CRV (and more particularly these MTC processing means) are configured to send in real time the position POSCrv of the headset CRV and the direction DIR0P of the gaze of the operator OP to the server SRVp

[0055] The camera CAM is fixed in the reference frame REF; it is configured to acquire images of the conveyor belt TR and of the objects moved by this belt and to send a video stream FV comprising these images to the server SRVp

[0056] In the embodiment described here, the SRVi server comprises an MDO module for detecting the images IMG0Bji of the objects OBJ; contained in the images of the video stream FV.

[0057] In the embodiment described here, the SRVi server comprises a position determination module MDP configured to determine, at a time h, the position POSOBJi, in the reference frame REF, of the objects OBJ; detected in the images of the video stream FV.

[0058] In the embodiment described here, the SRVi server comprises an MDI module of determination of interaction duration to measure the duration DI during which the operator OP looks at the same given object OBJ;*. It is considered in fact that the fact for the operator to look at an object OBJ*; constitutes, within the meaning of the present disclosure, an interaction of the operator with this object.

[0059] For this purpose, the interaction duration determination module MDI determines whether, at a given time f, the operator is looking at an object OBJ*; using, on the one hand, the position of the objects OBJ; detected in the images of the video stream FV and, on the other hand, the direction DIRop of the gaze of the operator OP at this time L. In the embodiment described here, the object detection module MDO sends, to the virtual reality headset CRV, the images IMG0Bji of the detected objects so that they are projected onto a screen of this headset.

[0060] The image IMG0Bji of the object OBJ*; being viewed by the operator OP can be presented on the screen highlighted or for example surrounded by a line, so that the operator can locate this object more easily.

[0061] In the embodiment described here, the server SRVi comprises a speed control module MCV configured to control the speed of the conveyor belt TR (or more generally the speed of an object moving device) as a function of this interaction duration.

[0062] In the embodiment described here, the MCV module sends, to the treadmill TR: - a signal SIG- to slow down the treadmill TR when the interaction duration DI is less than a first threshold dmin; - a SIG+ signal to accelerate the TR belt when the interaction duration DI is greater than a second threshold dmax, greater than the first threshold dmin.

[0063] The SYS system thus makes it possible to adjust the speed of the conveyor belt so that the operator OP looks at a given object OBJ*; for an interaction duration between dmin and d max*

[0064] In one embodiment, these thresholds are chosen so that the interaction duration DI must be sufficient for the operator and the minimum so as not to slow down the chain too much. These thresholds can be configured according to the type of part to be inspected and / or the operator. In another embodiment, the percentage of objects viewed by the operator during an interaction duration between dmin and dmax is counted and the control module MCV accelerates / slows down the speed of movement of the belt according to this percentage.

[0065] The present disclosure thus makes it possible to adapt the speed of movement of the treadmill according to the degree of fatigue and the concentration capacity of the operator.

[0066] In one embodiment, when an object has been viewed by the operator for a duration DI greater than the first threshold dmin, the color of the image of this projected object on the virtual reality headset screen is changed.

[0067] [Fig.3] shows an SRV2 server that can be used in another embodiment of the present disclosure.

[0068] The SRV2 server is remarkable in that the object detection module MDO P is configured to identify, in a database BD, a digital twin JN; of an object OBJi detected in an image.

[0069] In this embodiment, it is considered that the operator is looking at an object OBJ; if the direction DIR0P of his gaze crosses a volume, for example a cube or a sphere, encompassing the digital twin JN; of this object.

[0070] In this embodiment, the object detection module MDO sends to the virtual reality headset CRV, so that they are projected onto the screen of this headset, either the images IMGJNi of the detected objects or the images of their digital twins JN;.

[0071] [Fig.4] shows an SRV3 server that can be used in another embodiment of the present disclosure.

[0072] The SRV3 server is similar to the SRV2 server but it also includes a fault detection MDD module.

[0073] This MDD module is configured to analyze the images IMG0BJi of the objects OBJ; detected by the object detection module MDO in order to determine whether these objects have defects. This MDD module uses for example an artificial intelligence algorithm trained from images of objects having defects and images of objects not having defects.

[0074] In one embodiment, only images of objects detected as having a defect (or images of digital twins of these objects) are sent to the virtual reality headset to be projected to the operator, so that the operator is encouraged to only check the defective objects.

[0075] In one embodiment, only the images of the objects detected as not having a defect (or the images of the digital twins of these objects) are sent to the virtual reality headset to be projected to the operator, so that the operator is encouraged to only check the objects assumed to be perfect.

[0076] In another embodiment, criteria other than the presence of a defect may be used to determine the parts that the operator must check.

[0077] [Fig.5] represents the hardware architecture of a server SRVi, SRV2, SRV3 as described previously and which can be used in a system for controlling the speed of an object moving device in accordance with the present disclosure.

[0078] Such a server comprises in particular a processor 10, a RAM 11, a ROM 12 and communication means 13.

[0079] The read-only memory 12 constitutes a recording medium within the meaning of the present disclosure. It comprises a computer program PG in accordance with the present di- popularization.

[0080] This computer program PG comprises instructions for executing the steps of a method for controlling the speed of an object moving device when said program is executed by a computer.

[0081] The PG program defines functional modules of the SRVi, SRV2 or SRV3 server, which rely on or control the hardware elements 10, 11, 13 mentioned above, and for example:

[0082] - an MDO module for detecting images IMG0Bji of objects OBJ; contained in a FV video stream;

[0083] - a position determination MDP module configured to determine, at a instant f, the position POSOBJi, in the REF reference frame, of the objects OBJ; detected in the images of the FV video stream;

[0084] - an MDI interaction duration determination module for measuring the DI duration during which the operator OP looks at or manipulates a given object OBJ;*

[0085] - an MCV module for controlling the speed of an object moving device;

[0086] - a fault detection MDD module.

[0087] [Fig.6] represents the main steps of a speed control method of an object moving device according to a particular embodiment of the present disclosure.

[0088] In the embodiment described here, the method comprises an initialization step E5. In the embodiment described here, this initialization step comprises an initialization of the speed V0 of an object moving device OBJ, for example a treadmill TR and a determination of the position POSCrv of a virtual reality headset CRV worn by an operator OP.

[0089] In the embodiment described here, the method then comprises a loop of steps E10 to E50.

[0090] In the embodiment described here, the method comprises step E10 of determining the direction DIR0P of the gaze of the operator OP at a current time L

[0091] This DIR0P determination can for example be carried out using a CAMc camera incorporated in the virtual reality headset and directed towards the eyes of the operator OP.

[0092] In the embodiment described here, the detection method comprises a step E15 of acquiring images of the objects OBJ; moving on the conveyor belt TR at the current time L

[0093] In the embodiment described here, these images are acquired by a CAM camera and an FV video stream is created from these images.

[0094] In the embodiment described here, the detection method comprises a step E20 of detecting the images IMG0BJi of the objects OBJ; contained in the images of the stream FV video.

[0095] In the embodiment described here, the detection method comprises a step E25 of identifying a digital twin JN; of an object OBJi detected in an image.

[0096] In the embodiment described here, the detection method comprises a step E30 of determining the position POSOij of the objects OBJ; detected in the images of the video stream FV.

[0097] In the embodiment described here, the detection method comprises a step E35 of analyzing the images IMG0Bji of the objects OBJ; detected in the video stream FV to determine whether or not these objects OBJ; have defects.

[0098] In the embodiment described here, the detection method comprises a step E40 of projecting onto the screen of the virtual reality headset the images IMG0Bji of the objects detected in the video stream FV or the digital twins JN; of these objects. For example, the digital twins of the objects detected with defects are presented with a first color and the objects detected without defects are presented with a second color.

[0099] In the embodiment described here, the detection method comprises a step E45 of determining the duration DI during which the operator OP looks at an object OBJ*; given by using on the one hand the position POSOi of the objects OBJ; detected in the images of the video stream FV and on the other hand, the direction DIR0P of the gaze of the operator OP at the same instant. For example, it can be considered that the operator looks at an object OBJ; if the direction DIR0P of his gaze crosses a volume, for example a cube or a sphere, encompassing the digital twin JN; of this object.

[0100] In the embodiment described here, the detection method comprises a step E50 of controlling the speed of the conveyor belt TR (or more generally the speed of an object moving device) as a function of the interaction duration DI.

[0101] In the embodiment described here, this control consists of slowing down the object moving device if the interaction duration DI is less than a first threshold and accelerating it if the interaction duration DI is greater than a second threshold greater than the first threshold.

[0102] In another embodiment, the control is performed based on the rate of objects inspected relative to all objects on the belt, possibly determined using the digital twins of these objects.

[0103] In the embodiments described above, the direction of the operator's gaze is determined by the CAMc camera of the virtual reality headset. Alternatively, it can be determined by the position and orientation of the operator's head.

[0104] In another embodiment, the direction of the operator's gaze is completed by the CAM camera external to the virtual reality headset, which acquires images of the objects on the mat and produces the video stream.

[0105] In one embodiment, the operator does not need to wear a virtual reality headset, the position and direction of his head being able to be determined from tags arranged on a simple headband worn by the operator and a device configured to locate these tags.

[0106] In another embodiment, the objects themselves are located in the REF repository from labels arranged on these objects.

[0107] In another embodiment, the detection of an interaction does not consist of detecting that the operator is looking at the object but that the operator is manipulating the object (by detecting the presence of the image of the operator's hands on the image of the object) and the interaction duration DI is this manipulation duration.

[0108] In another embodiment, the detection of an interaction consists of detecting that the object is moving relative to the conveyor belt TR (necessarily because this object is grasped by the operator) the interaction duration DI is the duration during which this object is moved relative to the conveyor belt TR. This detection can also be done by image processing but also by mechanical means (detection of the weight of the object on the conveyor belt for example).

[0109] In the preceding description, we simply considered the overall duration of interaction with an object to adjust the speed of the movement device. Other more detailed rules can be envisaged, for example a rule verifying that the operator has examined each face of the object.

[0110] According to another example, a rule can verify that a set of portions (or points of interest) of an object have been examined (visually or by manipulation) by the operator. This may for example be portion(s) whose location on an object has been defined by configuration. In such embodiments, the MDI interaction duration determination module introduced above can verify the occurrence of an interaction by the operator for each of these portions (an absence of occurrence of an interaction for a portion considered resulting in a zero interaction duration for the portion considered) and, upon the occurrence of an interaction with one of these portions, measure the duration during which the operator looks at this portion of the object (the interaction duration then being relative to this portion) and the speed control module can take into account at least one of the interaction durations relative to the defined portions of an object.For example, the speed control module can take into account each of the interaction durations relating to the portions to be examined for an object).

Claims

Claims

1. Method for controlling the speed of a device (TR) for moving objects (OBJ;) as a function of at least one occurrence of interaction of an operator (OP) with at least one portion to be inspected of at least one of said objects.

2. Control method according to claim 1 wherein said at least one interaction occurrence is determined by a duration of interaction of said operator with said at least one portion of said at least one object.

3. Control method according to claim 2 wherein said duration is a duration of observation or manipulation of said at least one object by the operator.

4. Control method according to claim 2 or 3 comprising the following steps: - determination (E10) of a direction (DIR0P) of the gaze of the operator (OP); - detection (E20), in a video stream (FV) of at least one image (IMG oBji) of one of said moving objects; - determination (E30) of a position (POSOi) of said object (OBJ;); - determination (E45) of an interaction duration (DI) during which the operator (OP) looks at said object (OBJ;) from the position (POSOi) of said object (OBJ;) and the direction (DIR0P) of the gaze of the operator (OP); - control (E50) of a speed of movement of said device (TR) for moving objects (OBJ;) as a function of said duration (DI).

5. Control method according to claim 4 comprising a step (E25) of identifying a digital twin (JNi) of said object (OBJ;) detected in said flow, said interaction duration (DI) being a duration during which the direction (DIR0P) of the gaze of the operator (OP) crosses a volume encompassing said digital twin (JNi) of this object.

6. Control method according to claim 4 or 5 comprising a step of projection (E40), on the screen of a virtual reality headset of the operator (OP), of an image (IMG0Bji, IMGJNi) obtained from said image (IMG0Bji) detected in the video stream (FV).

7. Control method according to any one of claims 4 to 6 comprising an analysis (E35) of said at least one image (IMG0Bji) to determine whether or not said object (OBJ;) has a defect.

8. Control method according to claims 6 and 7 characterized in that the image (IMG0Bji, IMGJNi) projected on the screen of a virtual reality headset of the operator (OP) depends on the result of said analysis.

9. A computer program (PG) comprising instructions which, when the program is executed by a computer, cause the latter to implement a control method according to any one of claims 1 to 8.

10. A computer-readable recording medium on which a computer program (PG) according to claim 9 is recorded.

11. System (SYS) for controlling the speed of a device (TR) for moving objects (OBJ;) as a function of at least one occurrence of interaction of an operator (OP) with at least one portion to be inspected of at least one of said objects.

12. Control system (SYS) according to claim 10 wherein said device (TR) for moving objects (OBJ;) is a conveyor belt.

Images