System and method for detecting, tracking and projecting at least one object.

Abstract

System and method for detecting, tracking, and projecting the trajectory of an object. The invention relates primarily to a system for detecting and tracking the trajectory of an object moving in a public or private space, comprising computer means configured to determine the coordinates of the object's trajectory in an image frame, transpose these coordinates into the camera frame and then into the Earth's reference frame, and project the object's trajectory onto a plane bounded by two axes of the Earth's reference frame corresponding to the ground of the location. Figure to be published with the abstract: Fig. 1

Description

Title of the invention: System and method for detecting, tracking and projecting at least one object. technical field

[0001] The invention falls within the field of computer vision, and more specifically within the field of recognition and tracking of the movement of inanimate objects and / or individuals in public or private places. PRIOR ART AND DISADVANTAGES OF PRIOR ART

[0002] To assist people with disabilities, and in particular visually impaired people, means are planned to disseminate specific information to facilitate the orientation of these people wishing to board public transport vehicles.

[0003] It is known, via camera-type vision systems connected to computer means implementing computer vision algorithms, to identify static or moving objects and individuals and to follow their trajectories.

[0004] One drawback of these prior art devices lies in their transposition into the real world. Indeed, even if these computer vision systems make it possible to detect objects and individuals as well as their trajectories, it remains difficult to locate these objects / individuals in the place itself. OBJECTIVE OF THE INVENTION

[0005] The invention therefore aims to provide a system and a method which facilitate the location of objects and individuals in public or private places. Description of the invention

[0006] To this end, the invention relates to a system for detecting, tracking and projecting the trajectory of an object present in a place, particularly a public place, comprising at least one video camera and computer means for locating the object and tracking its trajectory, the computer means being configured, when the camera is oriented in a specific area of ​​the place and is in image acquisition mode, to: • Detect an object in the camera's field of vision; • Generate a bounding box for the object; • Determine, in a two-axis coordinate system associated with the acquired images, at least one coordinate of the bounding box of the object for each image acquired by the camera, this or these coordinates forming those of the trajectory of the object in the frame associated with the images; • Determine the coordinates of the trajectory in a three-axis frame, associated with the camera, from optical parameters of said camera and the coordinates of the trajectory in the frame associated with the images; • Determine the coordinates of the trajectory in a three-axis orthogonal frame, associated with the Earth's frame of reference, from the camera's position in this Earth's frame of reference and the coordinates of the trajectory in the frame associated with the camera, and • Project the trajectory into a plane defined by two of the axes of the terrestrial reference frame, which plane corresponds to the ground of the place.

[0007] The system may also include the following optional features considered individually or in all possible technical combinations: • The computer systems are configured to classify the detected object, specifically as an individual. • The computer resources include means of visualizing the trajectory of the object projected onto the ground of the location, in the terrestrial reference frame. • The control means are configured to generate the bounding box of the object in a quadrilateral shape, which box includes a bottom line whose coordinate along a vertical axis of the frame associated with the ground in the terrestrial reference frame is equal to zero.

[0008] The invention also relates to a method for detecting and tracking the trajectory of an object present in a place, particularly a public place, with a detection and tracking system comprising at least one video camera oriented in a determined area of ​​the place and which is in an image acquisition mode and computer means for locating the object and determining its trajectory, according to the following steps: • Detection of an object in the camera's field of vision; • Generation of a bounding box of the object; • Determination, in a two-axis coordinate system, associated with the acquired images, at least one coordinate of the bounding box of the object for each image acquired by the camera, this or these coordinates forming those of the trajectory of the individual in the frame associated with the images; • Determination of the coordinates of the trajectory in a three-axis frame, associated with the camera, from optical parameters of said camera and the coordinates of the trajectory in the frame associated with the images; • Determination of the trajectory coordinates in a three-axis orthogonal frame, associated with the Earth's reference frame, based on the camera's position in this Earth's reference frame and the trajectory coordinates in the frame associated with the camera, and • Projection of the trajectory into a plane defined by two of the axes of the terrestrial reference frame, which plane corresponds to the ground of the place.

[0009] The process may also include the following optional features considered individually or in all possible technical combinations: • The process includes a step of classifying the detected object. • The process includes a step of visualizing the object's trajectory projected onto the ground of the place, in the terrestrial frame of reference. • The bounding box of the object is quadrilateral in shape, which box includes a lower line whose coordinate along a vertical axis of the frame associated with the ground in the terrestrial reference frame is equal to zero. • The step of determining the coordinates of the trajectory in the coordinate system associated with the images includes the following steps: • Determination of at least one coordinate of the bounding box of the individual in the image n, n being a natural number greater than 1; • Searching over the n-1 previous images of a trajectory of said object; • Calculation of predictive coordinates of the bounding box in image n from the coordinates of the trajectory on the previous n-1 images; • Determination of the difference between the predicted coordinates and the determined coordinates of the bounding box to the image n; • If the difference is less than a determined threshold, the determined coordinates of the bounding box are assigned to the image n on the trajectory of said object for the n-1 previous images; • If the difference is greater than the determined threshold, a new trajectory is generated for the bounding box of said object. • The camera includes a digital sensor, and the coordinate system associated with said camera is orthogonal and includes two axes perpendicular to each other in the plane of the sensor and one axis perpendicular to the plane of the sensor. • The determination of the coordinate(s) of the bounding box of the object from the frame associated with the acquired images to the frame associated with the camera is implemented by an intrinsic transformation matrix taking into account the focal length of the camera, the optical center of the camera and the pixel size of the camera sensor. • Determining the coordinate(s) of the bounding box of the object from the camera's associated coordinate system to the coordinate system associated with the reference frame terrestrial is implemented with an extrinsic transformation matrix taking into account the spatial position and orientation of the camera sensor relative to the terrestrial reference frame. PRESENTATION OF THE FIGURES

[0010] Other features and advantages of the invention will become clear from the description given below, by way of example and not limitation, with reference to the attached figure:

[0011] [Fig. 1] The [Fig. 1] represents a flowchart illustrating the steps of the process of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] It is specified that the figure essentially represents one embodiment of the object of the invention but that there may be other embodiments which meet the definition of the invention.

[0013] The invention relates to a system and a method for determining, tracking, and projecting the trajectory of an object onto the ground of a public or private place, for example, but not limited to, a railway station. The invention also has application in a private place, for example, a warehouse or a store. The term "object" is understood by those skilled in the art to mean either an inanimate object—such as a suitcase—or a animate object, such as an animal or a person. In the following description, the term "person" will be used, given that the invention also applies to any type of detected object.

[0014] Such a system comprises at least one video camera-type vision device, and preferably several video cameras. Each of these cameras conventionally comprises a digital sensor of the CMOS or CCD type, which sensor is characterized by an array of cells commonly called pixels and defining the resolution of said sensor. Each camera also comprises a lens defined by its focal length f and by its optical center.

[0015] The system of the invention also includes means for visualizing the trajectory of an individual on the ground, in particular a screen, and computer means for locating and tracking the individual and determining its trajectory. These computer means also allow the vision device(s) and trajectory visualization means to be controlled. Furthermore, these computer means are configured to implement the method of the invention, the steps of which will be described with reference to [Fig. 1].

[0016] Firstly, a person skilled in the art understands that the process is implemented by computing means of said computer means, implementing one or more algorithms stored in a memory space of said computer means.

[0017] As part of the implementation of the method, the camera is directed towards a determined area of ​​the place, and performs a periodic acquisition of images, and where appropriate a periodic recording in the memory space of the computer means.

[0018] In a first step E1 of the process, the computer means detect an object in the camera's field of view. This object may be static or moving. The computer means then generate E2 a bounding box—generally rectangular—that encompasses the detected moving object. The bounding box is assigned four coordinates—one per vertex—in a coordinate system associated with the acquired image. This coordinate system is two-dimensional and comprises two perpendicular axes x and y. Any point in the image is located by its coordinates (u, v) in the coordinate system (x, y).

[0019] This type of detection is well known and is performed by at least one deep learning algorithm, which also allows for the classification of a set of pixels representative of the identified object, whether static or moving. Optionally, the process implements the next steps if the object is classified as an "individual." This classification is determined by computer means by analyzing a confidence score, which must be greater than a predetermined threshold. As an example, this classification is performed by the algorithm after training it with "pedestrian" data.

[0020] In this optional step E3, if the object is classified as an individual, then the process implements a second step which consists of determining, in the image coordinate system, the coordinates of the individual's trajectory. To do this, the computer means implement the following substeps.

[0021] In a first sub-step, the computer means determine the coordinates of the vertices of the bounding box on an image n, n being a natural number greater than or equal to 1.

[0022] If n=l, then there is no previous image to image n. Under these conditions, the computer means generate a new trajectory whose first point is located on this image n.

[0023] During a second sub-step and if n is greater than 1, the computer means search on the n-1 previous images for a trajectory of said object.

[0024] In a third substep, the control means calculate predictive coordinates of the center of the bottom line of the bounding box in image n, from the coordinates of the trajectory on the previous n-1 images. The bottom line is the line that connects the two bottom vertices of the bounding box.

[0025] In a fourth substep, these predictive coordinates are compared with the actual coordinates of the bounding box by computer means.

[0026] If the difference between the predicted coordinates and the actual coordinates of the bounding box in image n is less than a determined threshold, then the coordinates of the bounding box in image n are associated with the trajectory of said individual identified for the previous n-1 images.

[0027] Conversely, if the difference is greater than the determined threshold, then the computer means generate a new trajectory for the bounding box of the individual, starting at image n.

[0028] At this stage of the process, the computer means have therefore determined E4 the coordinates of a trajectory of an individual in the frame associated with the images.

[0029] It is understood for those skilled in the art that the method is applicable to both a moving object or individual and a static object or individual. If the individual or object is static, then the trajectory is simply formed by a point whose coordinate(s) are identical in all the images acquired by the camera.

[0030] In a third step E5 of the method, the computer means determine the coordinates of the individual's trajectory in a three-axis frame associated with the camera from intrinsic parameters and the coordinates of the trajectory in the frame associated with the images. The intrinsic parameters are the optical parameters of the camera and the characteristics of the sensor, and in particular the focal length f of the camera, the pixel size of the sensor and the optical axis of the camera.

[0031] The camera's coordinate system is three-dimensional and comprises two axes perpendicular to each other Xc and Yc in the sensor plane, and an axis Zc perpendicular to the sensor plane.

[0032] The coordinates of the optical axis are (u0, v0) in the coordinate system associated with the images and are equal to Q _ x- - k °h ct height of a pixel of the sensor valent (i -L). ku' k /

[0033] The transformation of the individual's trajectory coordinates from the image frame to the camera frame is performed by applying—using computer software—an intrinsic transformation matrix that takes into account the intrinsic parameters mentioned above. Furthermore, in a frame associated with the Earth's reference frame, which is three-dimensional and comprises two mutually perpendicular axes Xw and Yw in the (Xw, Yw) plane of the local ground, and an axis Zw perpendicular to the local ground plane, the bottom line of the box lies in the ground plane, so the coordinate along the Zw axis is equal to 0. This zero coordinate for the Zw axis reduces the number of unknowns during the transformation of the trajectory coordinates from the image frame to the camera frame, and thus greatly facilitate calculations for computer systems, or even make the system of equations solvable.

[0034] Thus, for each coordinate (u, v) in the image frame, it is necessary to apply the following transformation to obtain the coordinates of the trajectory in the frame associated with the camera:

[0035] = m0 + ktJ~^, y. = +kvf-^, and 0 = aXc + fiYc + yZe + ô, or a, fi, y and ô are scalar values ​​determined by computer means, according to methods well known to the person skilled in the art in computer vision software, for example a method of determination by manipulation of 3D point cloud.

[0036] The operations to be applied by computer means to the coordinates of the trajectory in the image frame in order to transpose them into the camera frame are therefore as follows:

[0037] In a fourth step E6 of the method, the computer means determine the coordinates of the individual's trajectory in a three-axis frame associated with the Earth's reference frame, based on extrinsic parameters and the coordinates of the trajectory in the frame associated with the camera. The extrinsic parameters are the camera's position in this Earth's reference frame, and more specifically the sensor's orientation and the camera's position relative to this Earth's reference frame.

[0038] The frame of reference in the terrestrial reference frame is three-dimensional and comprises two axes perpendicular to each other Xw and Yw in the (Xw, Yw) plane of the ground at the location, and an axis Zw perpendicular to the plane of the ground at the location.

[0039] The transformation of the coordinates of the individual's trajectory from the camera's frame of reference to the frame associated with the Earth's reference frame is performed by applying—using computer software—an extrinsic transformation matrix that takes into account the extrinsic parameters mentioned above. Such extrinsic and intrinsic matrices are well known and used in computer vision software. The result is obtaining the coordinates of the trajectory of each individual present in the camera's field of view, in a frame of reference associated with the Earth's reference frame.

[0040] Advantageously, the transformation of the camera's frame of reference to the real frame associated with the Earth's reference frame is also feasible when the camera is in motion, for example in translational or rotational motion: the camera's movements are additional extrinsic parameters integrated in the extrinsic transformation matrix, to calculate the coordinates of the individual's trajectory in the terrestrial reference frame.

[0041] In a fifth step E7 of the process, the trajectory determined in the Earth's frame of reference is then projected onto the (Xw, Yw) plane of the ground in the public or private place in which the individual is moving. This yields a two-dimensional representation of the trajectory in the (Xw, Yw) plane of the ground in the Earth's frame of reference.

[0042] Finally, in a final step E8, the ground plane (Xw, Yw) and the trajectory projected onto said plane (Xw, Yw) are displayed on the visualization means. This representation thus provides a two-dimensional view of the trajectory projected onto the ground at the location, in the Earth's frame of reference. The invention therefore facilitates the tracking of individuals moving within the location.

Claims

Demands

1. A system for detecting and tracking the trajectory of an object present in a place, particularly a public place, comprising at least one video camera and computer means for locating the object and tracking its trajectory, the computer means being configured, when the camera is oriented in a determined area of ​​the place and is in an image acquisition mode, to: • Detect (E1) an object in the camera's field of view; • Generate (E2) a bounding box of the object; • Determine (E4), in a two-axis coordinate system associated with the acquired images, at least one coordinate of the bounding box of the object for each image acquired by the camera, this or these coordinates forming those of the object's trajectory in the coordinate system associated with the images;• Determine (E5) the coordinates of the trajectory in a three-axis frame associated with the camera from optical parameters of said camera and the coordinates of the trajectory in the frame associated with the images; • Determine (E6) the coordinates of the trajectory in a three-axis orthogonal frame associated with the Earth's reference frame from the camera's position in this Earth's reference frame and the coordinates of the trajectory in the frame associated with the camera, and • Project (E7) the trajectory into a plane defined by two of the axes of the Earth's reference frame, which plane corresponds to the ground of the location.

2. System according to the preceding claim, characterized in that the computer means are configured to classify (E3) the detected object, in particular as being an individual.

3. System according to claim 1 or 2, characterized in that the computer means include means for visualizing the trajectory of the object projected onto the ground of the place, in the terrestrial reference frame.

4. A system according to any one of the preceding claims, characterized in that the control means are configured to generate the bounding box of the object in a quadrilateral shape, which box includes a lower line whose coordinate along a vertical axis of the frame associated with the ground in the terrestrial reference frame is equal to zero.

5. A method for detecting and tracking the trajectory of an object present in a place, particularly a public place, with a detection and tracking system comprising at least one video camera oriented in a determined area of ​​the place and which is in an image acquisition mode and computer means for locating the object and determining its trajectory, according to the following steps: • Detection (E1) of an object in the camera's field of vision; • Generation (E2) of a bounding box of the object; • Determination (E4), in a two-axis coordinate system associated with the acquired images, of at least one coordinate of the bounding box of the object for each image acquired by the camera, this or these coordinates forming those of the trajectory of the individual in the coordinate system associated with the images;• Determination (E5) of the coordinates of the trajectory in a three-axis frame, associated with the camera, from optical parameters of said camera and the coordinates of the trajectory in the frame associated with the images; • Determination (E6) of the coordinates of the trajectory in a three-axis orthogonal frame, associated with the Earth's reference frame, from the camera's position in this Earth's reference frame and the coordinates of the trajectory in the frame associated with the camera, and • Projection (E7) of the trajectory into a plane defined by two of the axes of the Earth's reference frame, which plane corresponds to the ground of the location.

6. Method according to the preceding claim, characterized in that it comprises a classification step (E3) of the detected object.

7. Method according to claim 5 or 6, characterized in that it comprises a visualization step (E8) of the trajectory of the object projected onto the ground of the place, in the terrestrial reference frame.

8. A method according to any one of claims 5 to 7, characterized in that the bounding box of the object is quadrilateral in shape, which box comprises a lower line whose coordinate along a vertical axis of the coordinate system associated with the ground in the terrestrial reference frame is equal to zero.

9. A method according to any one of claims 5 to 8, characterized in that the step of determining (E4) the coordinates of the trajectory in the frame associated with the images comprises the steps of: • Determining at least one coordinate of the bounding box of the individual in image n, n being a natural number greater than 1; • Searching the n-1 previous images of a trajectory of said object; • Calculating predictive coordinates of the bounding box in image n from the coordinates of the trajectory in the n-1 previous images; • Determining the difference between the predictive coordinates and the determined coordinates of the bounding box in image n; • If the difference is less than a determined threshold, assigning the determined coordinates of the bounding box in image n to the trajectory of said object for the n-1 previous images;• If the difference is greater than the determined threshold, a new trajectory is generated for the bounding box of said object.

10. A method according to any one of claims 5 to 9, characterized in that the camera comprises a digital sensor, and in that the frame associated with said camera is an orthogonal frame comprising two axes perpendicular to each other in the plane of the sensor and one axis perpendicular to the plane of the sensor.

11. A method according to any one of claims 5 to 10, characterized in that the determination (E5) of the coordinate(s) of the bounding box of the object from the frame associated with the acquired images to the frame associated with the camera is implemented by an intrinsic transformation matrix taking into account the focal length of the camera, the optical center of the camera and the pixel size of the camera sensor.

12. A method according to any one of claims 5 to 11, characterized in that the determination (E6) of the coordinate(s) of the box The encompassing of the object from the frame associated with the camera to the frame associated with the terrestrial reference frame is implemented with an extrinsic transformation matrix that takes into account the spatial position and orientation of the camera sensor relative to the terrestrial reference frame.