Method and system for controlling the speed of a device for moving objects

Abstract

One embodiment of this method for controlling the speed of a device for moving objects (OBJi) comprises the following steps: - determining (E10) a direction (DIROP) of the gaze of an operator; - detecting (E20), in a video stream (FV), at least one image (IMGOBJ i) of one of the moving objects; - determining (E30) a position of the object (OBJi); - determining (E45) a duration (DI) of interaction during which the operator (OP) is looking at the object (OBJi) on the basis of the position (POSOi) of the object (OBJi) and the direction (DIROP) of the gaze of the operator; - controlling (E50) a movement speed of the device (TR) for moving objects (OBJi) according to the duration (DI).

Description

[0001] Description

[0002] Title of the invention: Method and system for controlling the speed of an object moving device

[0003] Prior art

[0004] This disclosure is in the general context of quality control of objects, for example at the end of a production line, for example in an industrial environment.

[0005] In industry, it is known that part verification and quality control tasks can be hampered by machine throughput, leading to verification errors and the production of non-conforming parts.

[0006] This disclosure aims at a solution to improve this situation.

[0007] Subject matter and summary of the invention

[0008] More specifically, the present disclosure relates to a method for controlling the speed of a device for moving objects based on at least one occurrence of interaction of an operator with at least one portion to be inspected of at least one of said objects.

[0009] Correlatively, the present disclosure relates to a system for controlling the speed of a device for moving objects based on at least one occurrence of interaction of an operator with at least one portion to be inspected of at least one of said objects.

[0010] The object moving device can be of any nature. For example, it can be a conveyor belt.

[0011] An operator may be a human or a device (hereinafter referred to as an operator device) configured to interact with the object(s) moved by the object moving device. Said operator device may comprise, for example, a camera, a motion sensor, a robot or a gripping device.

[0012] The operator may observe, examine, manipulate, or control the objects moved by the object moving device. These operator actions (observation, examination, manipulation, control, etc.) are examples of operator interaction with the object(s). The type of operator interaction with the object(s) varies, among other things, depending on the nature of the operator.

[0013] In the current state of the art, due to the large amount of information provided by the devices in a production line and the often constant speed of the conveyor belts, it is not possible to ensure that the operator has had the necessary time to examine all the parts.

[0014] The method and system of the present disclosure propose to adapt the speed of presentation of the objects to the user to ensure that the operator has been able to interact with the objects in order to be able to inspect or examine them.

[0015] In one embodiment, the occurrence of an interaction of the operator with at least one portion of the object is determined by measuring a duration of interaction of the operator with this portion of the object.

[0016] Thus, as objects scroll by, by measuring the duration of the operator's interaction with the objects or portions of these objects, the process makes it possible to check whether the operator has had time to verify the conformity of each object. As soon as an object has not had time to be checked, the process can slow down the conveyor belt or even stop it so that the operator can complete his task.

[0017] In one embodiment, the interaction duration is a duration of observation of said at least one object by the operator. In a particular embodiment, the method comprises the following steps:

[0018] - determination of the operator's gaze direction;

[0019] - detection, in a video stream, of at least one image of one of said moving objects;

[0020] - determination of a position of said object;

[0021] - determination of an interaction duration during which the operator looks at said object from the position of said object and the direction of the operator's gaze;

[0022] - controlling a speed of movement of said object moving device as a function of said duration.

[0023] Thus, the present disclosure proposes to use an eye-tracking method to measure the duration with which an operator observes an object or a portion of an object and to adapt the speed of presentation of the next objects to this operator according to this duration.

[0024] In this embodiment, the present disclosure more specifically proposes to track the eye movements of the operator to evaluate the objects or portions of objects that he has examined. By using eye tracking, the system ensures that the activity has been completely carried out, adjusting the speed of the machine accordingly. This method makes it possible to detect fatigue or the need to slow down the pace, taking into account the needs of the operator by tracking his gaze.

[0025] In one embodiment, the method comprises a step of identifying a digital twin of the object detected in the video stream, the interaction duration being a duration during which the direction of the operator's gaze crosses a volume encompassing the digital twin of this object.

[0026] This embodiment allows in particular to deport the calculation of the interaction duration to a machine and a dedicated process. In particular, this determination can be made at the time of the measurement of the tracking of the direction of the gaze in the headset, the latter knowing the digital twin of the room. It can also be deported to a server which has knowledge of both the direction of the gaze and the twin of the object.

[0027] This embodiment allows, by working on simple volumes, to greatly reduce the complexity of the algorithm used to determine whether the operator is actually looking at an object.

[0028] The embodiment of the present disclosure based on eye tracking, a technique for recording eye movements, to determine the direction of the operator's gaze is very precise and efficient.

[0029] It uses precise cameras and powerful algorithms to accurately measure eye position and movement.

[0030] In another embodiment, the interaction time is a duration of manipulation of the object by the operator. For this purpose, simpler visual means can be used to detect the presence of hands on the object, or even mechanical means to detect that the object has been lifted, by measuring the weight on the mat for example.

[0031] In one embodiment, the control method, the subject of this disclosure, comprises an analysis of the images of the video stream to determine whether or not an object detected in this stream has a defect.

[0032] This analysis can, for example, be implemented by an artificial intelligence algorithm trained on images of objects with defects and images of objects without defects.

[0033] The integration of artificial intelligence into factories introduces a margin of error, requiring human validation to confirm or reject results.

[0034] The present disclosure proposes to carry out this validation by human inspection of the objects, the validation rate being adjusted according to the operator's ability to look at an object for a sufficient length of time, this ability being likely to vary over time, in particular according to his state of concentration or fatigue.

[0035] In one embodiment, the method comprises a step of projecting, onto the screen of a virtual reality headset of the operator, an image obtained from said image detected in the video stream. This mechanism advantageously makes it possible to encourage the user to examine a particular part by making it appear, for example, highlighted on the screen.

[0036] In one embodiment, the image projected onto the operator's virtual reality headset screen depends on the result of the fault analysis.

[0037] Advantageously, only parts with a defect, or on the contrary only parts assumed to be perfect, can be projected onto the operator's helmet.

[0038] In another embodiment, the projected pieces are randomly selected assumed perfect pieces.

[0039] Alternatively, depending on whether a defect is detected or not, the images of the objects can be projected differently, for example in different colors.

[0040] The present disclosure also relates to a computer program comprising instructions for executing the steps of the method for controlling the speed of an object moving device when said program is executed by a computer.

[0041] This program may use any programming language, and may be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0042] The present disclosure also relates to a computer-readable information carrier comprising instructions of a computer program as mentioned above. The information carrier may be any entity or device capable of storing the program. For example, the carrier may comprise a storage means, such as a ROM, a non-volatile memory of the flash type or a magnetic recording means, for example a hard disk. Furthermore, the information carrier may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The program according to the present disclosure may in particular be downloaded from a network such as the Internet.Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to perform or to be used in the performance of the method in question.

[0043] Brief description of the drawings:

[0044] Other characteristics and advantages of the present disclosure will emerge from the description given below, with reference to the appended drawings which illustrate exemplary embodiments thereof without any limiting character. In the figures:

[0045] - Figure 1 shows an operator in front of an example of an object moving device; Figure 2 shows a speed control system of an object moving device according to an embodiment of the present disclosure

[0046] - Figure 3 shows a server that can be used in a control system according to another embodiment of the present disclosure;

[0047] - Figure 4 shows a server that can be used in a control system according to another embodiment of the present disclosure;

[0048] - Figure 5 shows the hardware architecture of a server that can be used in a control system according to an embodiment of the present disclosure; and

[0049] - Figure 6 represents in flowchart form the main steps of a method for controlling the speed of a device for moving objects in accordance with a particular embodiment of the present disclosure.

[0050] Description of the embodiments

[0051] Figure 1 shows an operator OP in front of a conveyor belt TR configured to drive objects OBT.

[0052] In the embodiment described here, it is assumed that the treadmill is driven by two rollers mounted on a fixed frame and that the operator is static relative to a reference frame REF.

[0053] The objects OBT therefore move in front of the operator OP and the conveyor belt TR constitutes an object moving device within the meaning of the present disclosure.

[0054] In this example, the OP operator is equipped with a CRV virtual reality headset.

[0055] In the embodiment described here, the virtual reality headset CRV is provided with a camera CAM c and processing means (processor, memories, computer program) MTc configured to determine: the position (and orientations) POSCRV of the CRV helmet; and

[0056] - the DIR management 0P from the perspective of the OP operator in the REF reference system.

[0057] To simplify the description, below, the POSCRV position is here a data structure which includes both the coordinates of the CRV helmet and its orientation according to the three orthonormal dimensions of the REF reference frame. It will be assumed in the embodiment described here that the direction DIR 0P of the gaze of the operator OP is obtained by determining an average direction calculated from the gaze directions of each of the eyes of the operator OP.

[0058] Figure 2 represents a control system SYS according to a particular embodiment.

[0059] In the embodiment described here, the SYS control system comprises:

[0060] - the CRV virtual reality headset,

[0061] - the TR conveyor belt (object moving device); a CAM camera; and

[0062] - an SRVi server, these elements being interconnected between them by an unreferenced network.

[0063] In the embodiment of Figure 2, the virtual reality headset CRV (and more particularly these processing means MT C ) are configured to send in real time the POSCRV position of the CRV helmet and the direction DIR 0P from the OP operator's view to the SRVi server.

[0064] The CAM camera is fixed in the REF reference frame; it is configured to acquire images of the TR conveyor belt and the objects moved by this belt and to send a FV video stream containing these images to the SRVi server.

[0065] In the embodiment described here, the SRVi server comprises an MDO module for detecting IMGOBJI images of OBT objects contained in the images of the FV video stream.

[0066] In the embodiment described here, the SRVi server comprises a position determination module MDP configured to determine, at a time ti, the position POSOBJI, in the REF reference frame, of the OBT objects detected in the images of the FV video stream.

[0067] In the embodiment described here, the SRVi server comprises an MDI module for determining the interaction duration to measure the duration DI during which the operator OP looks at the same given object OBT*. It is considered that the fact that the operator looks at an object OBJ*i constitutes, within the meaning of the present disclosure, an interaction of the operator with this object.

[0068] For this purpose, the MDI interaction duration determination module determines whether, at a time ti, the operator is looking at a given object OBJ*i using, on the one hand, the position of the objects OBJi detected in the images of the FV video stream and, on the other hand, the direction DIR 0Pof the gaze of the operator OP at this instant ti. In the embodiment described here, the object detection module MDO sends, to the virtual reality headset CRV, the images IMGOBJI of the detected objects so that they are projected onto a screen of this headset. The image IMGOBJI of the object OBJ*i currently being viewed by the operator OP can be presented on the screen highlighted or for example surrounded by a line, so that the operator can more easily locate this object.

[0069] In the embodiment described here, the SRVi server comprises a speed control module MCV configured to control the speed of the treadmill TR (or more generally the speed of an object moving device) as a function of this interaction duration.

[0070] In the embodiment described here, the MCV module sends, to the TR conveyor belt:

[0071] - a SIG- signal to slow down the TR belt when the interaction duration DI is lower than a first threshold dmin;

[0072] - a SIG+ signal to accelerate the TR belt when the interaction duration DI is greater than a second threshold dmax, greater than the first threshold dmin.

[0073] The SYS system thus makes it possible to adjust the speed of the conveyor belt so that the operator OP looks at a given object OBJ*i for an interaction duration between dmin and dmax.

[0074] In one embodiment, these thresholds are chosen so that the interaction duration DI must be sufficient for the operator and minimal to not slow down the chain too much. These thresholds can be configured according to the type of part to be inspected and / or the operator. In another embodiment, the percentage of objects viewed by the operator during an interaction duration between dmin and dmax is counted and the MCV control module accelerates / slows down the speed of movement of the belt according to this percentage.

[0075] The present disclosure thus makes it possible to adapt the speed of movement of the treadmill according to the degree of fatigue and the concentration capacity of the operator.

[0076] In one embodiment, when an object has been viewed by the operator for a duration DI greater than the first threshold dmin, the color of the image of this object projected on the screen of the virtual reality headset is modified.

[0077] Figure 3 shows an SRV2 server that may be used in another embodiment of the present disclosure.

[0078] The SRV2 server is remarkable in that the MDO object detection module P is configured to identify, in a database BD, a digital twin JN, of an object OBJi detected in an image.

[0079] In this embodiment, the operator is considered to be looking at an object OBJi if the direction DIR 0Pof his gaze crosses a volume, for example a cube or a sphere, encompassing the digital twin JN, of this object. In this embodiment, the object detection module MDO sends to the virtual reality headset CRV, so that they are projected on the screen of this headset, either the images IMGjNi of the detected objects or the images of their digital twins JNj.

[0080] Figure 4 shows an SRV3 server that may be used in another embodiment of the present disclosure.

[0081] The SRV3 server is similar to the SRV2 server but additionally includes an MDD fault detection module.

[0082] This MDD module is configured to analyze the IMGOBJI images of the OBJi objects detected by the MDO object detection module to determine whether these objects have defects. This MDD module uses, for example, an artificial intelligence algorithm trained on images of objects with defects and images of objects without defects.

[0083] In one embodiment, only images of objects detected as having a defect (or images of digital twins of those objects) are sent to the virtual reality headset to be projected to the operator, so that the operator is encouraged to only check the defective objects.

[0084] In one embodiment, only images of objects detected as having no defects (or images of digital twins of these objects) are sent to the virtual reality headset to be projected to the operator, so that the operator is encouraged to only inspect objects assumed to be perfect.

[0085] In another embodiment, criteria other than the presence of a defect may be used to determine which parts the operator should inspect.

[0086] Figure 5 represents the hardware architecture of a server SRVi, SRV2, SRV3 as previously described and which can be used in a system for controlling the speed of an object moving device in accordance with the present disclosure.

[0087] Such a server comprises in particular a processor 10, a RAM 11, a ROM 12 and communication means 13.

[0088] The read-only memory 12 constitutes a recording medium within the meaning of the present disclosure. It comprises a computer program PG in accordance with the present disclosure.

[0089] This PG computer program comprises instructions for performing the steps of a method of controlling the speed of an object moving device when said program is executed by a computer.

[0090] The PG program defines functional modules of the SRVi, SRV2 or SRV3 server, which rely on or control the hardware elements 10, 11, 13 mentioned above, and for example: - an MDO module for detecting the IMGOBJI images of the OBJi objects contained in an FV video stream;

[0091] - a position determination module MDP configured to determine, at a time ti, the position POSOBJI, in the reference frame REF, of the objects OBJi detected in the images of the video stream FV;

[0092] - an MDI interaction duration determination module to measure the duration DI during which the operator OP looks at or manipulates a given object OBJi*

[0093] - an MCV module for controlling the speed of an object moving device;

[0094] - an MDD fault detection module.

[0095] Figure 6 represents the main steps of a method for controlling the speed of an object moving device according to a particular embodiment of the present disclosure.

[0096] In the embodiment described here, the method comprises an initialization step E5. In the embodiment described here, this initialization step comprises an initialization of the speed VO of an object movement device OBJi, for example a treadmill TR and a determination of the position POSCRV of a virtual reality headset CRV worn by an operator OP.

[0097] In the embodiment described here, the method then comprises a loop of steps E10 to E50.

[0098] In the embodiment described here, the method comprises a step E10 of determining the direction DIR 0P from the view of the operator OP at a current time ti.

[0099] This DIR determination 0P can for example be achieved using a CAM camera c incorporated into the virtual reality headset and directed towards the eyes of the OP operator.

[0100] In the embodiment described here, the detection method comprises a step E15 of acquiring images of the objects OBJi moving on the conveyor belt TR at the current time ti.

[0101] In the embodiment described herein, these images are acquired by a CAM camera and an FV video stream is created from these images.

[0102] In the embodiment described here, the detection method comprises a step E20 of detecting the images IMGOBJÏ of the objects OBJi contained in the images of the video stream FV.

[0103] In the embodiment described here, the detection method comprises a step E25 of identifying a digital twin JNi of an object OBJi detected in an image.

[0104] In the embodiment described here, the detection method comprises a step E30 of determining the position POS Oi , objects OBJi detected in the images of the FV video stream. In the embodiment described here, the detection method comprises a step E35 of analyzing the images IMGOBJI of the objects OBJi detected in the FV video stream to determine whether or not these objects OBJi have defects.

[0105] In the embodiment described here, the detection method comprises a step E40 of projecting onto the screen of the virtual reality headset the images IMGOBJI of the objects detected in the video stream FV or the digital twins JN of these objects. For example, the digital twins of the objects detected with defects are presented with a first color and the objects detected without defects are presented with a second color.

[0106] In the embodiment described here, the detection method comprises a step E45 of determining the duration DI during which the operator OP looks at a given object OBJ*i using on the one hand the position POSoi of the objects OBJi detected in the images of the video stream FV and on the other hand, the direction DIR 0P of the operator's gaze OP at the same time. For example, we can consider that the operator is looking at an object OBJi if the direction DIR 0Phis gaze crosses a volume, for example a cube or a sphere, encompassing the digital twin JNi of this object.

[0107] In the embodiment described here, the detection method comprises a step E50 of controlling the speed of the conveyor belt TR (or more generally the speed of an object moving device) as a function of the interaction duration DI.

[0108] In the embodiment described here, this control consists of slowing down the object moving device if the interaction duration DI is less than a first threshold and speeding it up if the interaction duration DI is greater than a second threshold greater than the first threshold.

[0109] In another embodiment, the control is performed based on the ratio of objects inspected relative to all objects on the belt, possibly determined using digital twins of those objects.

[0110] In the embodiments described above, the direction of the operator's gaze is determined by the CAM camera c of the virtual reality headset. Alternatively, it can be determined by the position and orientation of the operator's head.

[0111] In another embodiment, the direction of the operator's gaze is determined by the CAM camera external to the virtual reality headset, which acquires images of the objects on the mat and produces the video stream.

[0112] In one embodiment, the operator does not need to wear a virtual reality headset, the position and direction of his head being able to be determined from labels arranged on a simple headband worn by the operator and a device configured to locate these labels (in English tag). In another embodiment, the objects themselves are located in the REF reference system from labels arranged on these objects.

[0113] In another embodiment, the detection of an interaction does not consist of detecting that the operator is looking at the object but that the operator is manipulating the object (by detecting the presence of the image of the operator's hands on the image of the object) and the interaction duration DI is this manipulation duration.

[0114] In another embodiment, the detection of an interaction consists of detecting that the object moves relative to the conveyor belt TR (necessarily because this object is grasped by the operator). The interaction duration DI is the duration during which this object is moved relative to the conveyor belt TR. This detection can also be done by image processing but also by mechanical means (detection of the weight of the object on the conveyor belt for example).

[0115] In the previous description, we simply considered the overall duration of interaction with an object to adjust the speed of the moving device. Other, more refined rules can be considered, for example a rule verifying that the operator has examined each face of the object.

[0116] According to another example, a rule can verify that a set of portions (or points of interest) of an object have been examined (visually or by manipulation) by the operator. These may for example be portion(s) whose location on an object has been defined by configuration. In such embodiments, the MDI interaction duration determination module introduced above can verify the occurrence of an interaction by the operator for each of these portions (an absence of occurrence of an interaction for a portion considered resulting in a zero interaction duration for the portion considered) and, upon the occurrence of an interaction with one of these portions, measure the duration during which the operator looks at this portion of the object (the interaction duration then being relative to this portion) and the speed control module can take into account at least one of the interaction durations relative to the defined portions of an object.For example, the speed control module can take into account each of the interaction durations relating to the portions to be examined for an object).

Claims

CLAIMS

1. Method for controlling the speed of a device (TR) for moving objects (OBJi) as a function of at least one occurrence of interaction of an operator (OP) with at least one portion of at least one of said objects.

2. A control method according to claim 1 wherein said at least one portion is a portion to be inspected by said operator (OP).

3. A control method according to claim 1 or claim 2 wherein said at least one interaction occurrence is determined by a duration of interaction of said operator with said at least one portion of said at least one object.

4. Control method according to claim 3 wherein said duration is a duration of observation or manipulation of said at least one object by the operator.

5. Control method according to claim 3 or 4 comprising the following steps: - determination (E10) of a direction (DIR 0P ) from the operator's (OP) perspective; - detection (E20), in a video stream (FV) of at least one image (IMGOBJD of one of said moving objects; - determination (E30) of a position (POS Oi ) of said object (OBJi); - determination (E45) of an interaction duration (DI) during which the operator (OP) looks at said object (OBJi) from the position (POS Oi ) of said object (OBJi) and of the direction (DIR 0P ) from the operator's (OP) perspective; - control (E50) of a speed of movement of said device (TR) for moving objects (OBJi) as a function of said duration (DI).

6. Control method according to claim 5 comprising a step (E25) of identifying a digital twin (JNi) of said object (OBJi) detected in said flow, said interaction duration (DI) being a duration during which the direction (DIR 0P ) of the operator's gaze (OP) crosses a volume encompassing said digital twin (JNi) of this object.

7. Control method according to claim 5 or 6 comprising a step of projection (E40), on the screen of a virtual reality headset of the operator (OP), of an image (IMGOBJI, IMGJND obtained from said image (IMGOBJI) detected in the video stream (FV).

8. Control method according to any one of claims 5 to 7 comprising an analysis (E35) of said at least one image (IMGOBJI) to determine whether or not said object (OBJj) has a defect.

9. Control method according to claims 7 and 8 in which the image (IMGOBJI, IMGJNI) projected on the screen of a virtual reality headset of the operator (OP) depends on the result of said analysis.

10. A computer program (PG) comprising instructions which, when the program is executed by a computer, cause the latter to implement a control method according to any one of claims 1 to 9.

11. A computer-readable recording medium on which a computer program (PG) according to claim 10 is recorded.

12. System (SYS) for controlling the speed of a device (TR) for moving objects (OBJ) as a function of at least one occurrence of interaction of an operator (OP) with at least a portion of at least one of said objects.

13. Control system (SYS) according to claim 12 wherein said at least one portion is a portion to be inspected by said operator (OP).

14. Control system (SYS) according to claim 12 or claim 13 wherein said device (TR) for moving objects (OBJ.) is a conveyor belt.

Images