Method and device for counting objects in a moving vehicle

Abstract

A method and apparatus for discontinuously counting objects (4) on a moving vehicle (1) moving in a continuous flow along a straight or substantially straight route, for example to count cylinders loaded on moving vehicles, such as vehicles entering and / or leaving an industrial site.

Description

[0001] The present invention relates to the field of counting and distinguishing passing objects or natural persons or even animals by means of video image analysis.

[0002] In this area of ​​technology, the existing art is large and varied, both in terms of what is counted, the context in which the counting takes place, and the methods employed.

[0003] The following documents are mentioned here by way of example: EP-3 005 231 A2, WO 09 / 004479, WO 22 / 076443 A1, US Pat. No. 10,559,091, US Pat. No. 10,769,808 and WO 17 / 123920 A1, or CN 113,763,433 A.

[0004] In the context of the present invention, of particular interest is the discontinuous counting of physical objects in a unidirectional continuous flow (i.e. moving in a substantially straight line), where, i.e. in the present case, such a continuous flow of physical objects is obtained in an industrial environment such as a factory, a cylinder filling center or even a cylinder distribution center, where a flow of entering and leaving vehicles transporting cylinders is observed, the flow possibly entering or leaving via the same site / location / site entrance or indeed via different site entrances, the present invention allowing the objects in question to be counted and, where appropriate, also the number of counted objects to be classified into determined subcategories, such as cylinder color (and therefore gas type) or even cylinders of different types of caps, or even cylinders of different diameters, etc.

[0005] Thus, as will be seen in more detail hereinafter, the present invention aims at providing a method and apparatus for counting gas cylinders entering or leaving a site, whether they are full or empty, of different sizes and colors, or equipped with various forms of cylinder caps, which method is automated and improves the reliability of the prior art.

[0006] As described above, the present invention relates to discontinuous counting of physical objects that flow continuously in one direction, so the terms used in this technical field will now be explained:

[0007] - A stream is called continuous when the object of interest present therein does not stop during the camera acquisition phase. In the context of the present invention, this is the case where the truck is not stopped under the camera, so the camera only gets a partial view of the truck's load in each of the component images of the video. Even if the camera is able to see the entire load of the vehicle and / or the vehicle is stopped intentionally or unintentionally, and / or the path followed by the vehicle can be of any nature, as will be seen, the present invention remains robust and applicable under these image capture conditions.

[0008] -If the object stops at each acquisition and then continues to move, the flow is said to be discontinuous.

[0009] - If the object of interest is counting in groups rather than cumulatively over a given period of time, the count is considered discontinuous. In the context of the present invention, the object of interest is counting the number of gas cylinders per vehicle.

[0010] - Counts are considered continuous when there is no "grouping" of objects of interest, such as customers entering or leaving a commercial building.

[0011] It is known in the literature to count discontinuous flows discontinuously, the field of the sensor then comprising the entire counting area, which remains stationary for a short time in order to allow a complete image to be acquired by a dedicated camera and to be analyzed by suitable algorithms.

[0012] It is also known to count continuously in a continuous flow and then seek to estimate the change of the indicator, the concept of unit or group of objects not being applicable. Mention may be made here of the example given above of counting customers entering a commercial property.

[0013] It is therefore necessary to provide a new counting method to solve the problem faced here, where the object of interest is not the vehicle but the object transported by the vehicle, such as gas cylinders, and therefore an attempt is made to count the number of gas cylinders per truck under the following conditions: the truck does not stop to allow the capture of images - it passes without stopping under a gantry equipped with image capture devices ("continuous flow") - and the field of view of the camera located on the gantry is necessarily not large enough to cover the entire load of the truck (the counting is therefore discontinuous, as it is necessary to analyze the video and separate the sequences of interest corresponding to the passage of the truck).

[0014] It is worth noting that if another comparison method is used (not the method of the present invention), in which the count is continuous, then when multiple vehicles pass in a given time period, the number of cylinders will be counted and summed.

[0015] It will be apparent to those skilled in the art that the nature of the method selected by the present invention and the technical problems arising therefrom are as follows:

[0016] The vehicles of interest are visually distinguishable from other elements or vehicles that may pass through the area in question (such as cars, tank trucks, bicycles, pedestrians, forklifts, etc.). However, in this example, despite the fact that the two vehicles of interest are of the same nature, it must be possible to distinguish vehicle A, which transports gas cylinders, from another vehicle B, which also transports gas cylinders (for example, certain logistic flows must be excluded from the count for some reason X or Y). This distinction cannot be made naturally, either based on the appearance of these vehicles or on the content of their load. However, the number of counted objects must be compared with the "valid" number contained in the information system;

[0017] The length of the vehicles transporting gas cylinders is usually very long, the camera’s field of view cannot cover the entire load, and a single image captured is not enough to count all the cylinders, so video streaming is required;

[0018] ● Regarding the previous point, although the camera can certainly be installed at a higher height, this may be incompatible with the laws and regulations in force in some countries (installation standards, safety, etc.);

[0019] ● Manual counting is neither systematic nor reliable. Therefore, this method cannot be based on trust in people. Factors that may reduce reliability include non-compliance with instructions, weather, light levels, etc.

[0020] - The counting method must allow for overlapping objects of interest due to the temporal component of the video stream.

[0021] The purpose of the present invention is to allow the following to be achieved:

[0022] ● Regular counting with human involvement through technical solutions to allow manual counting in case of large deviations from the target value, thus allowing errors to be corrected;

[0023] ● Processing of continuous flows for discontinuous counting, systematically and independently of human factors, which allows to preserve the properties and configuration of the existing flows, thus achieving transparency for the user (implementation of the method according to the invention does not require closing the site, re-examination of the design of the plant or process, redefinition of the tasks of each operator, etc.).

[0024] The invention thus relates to a method for discretely counting objects present on a moving vehicle, said movement being in a continuous stream along a straight or substantially straight path, said vehicle being considered as a vehicle of interest capable of being identified with respect to other vehicles of the same or different nature, for example for counting gas cylinders loaded on a moving vehicle, such as vehicles entering and / or leaving an industrial site, wherein:

[0025] a. providing at least one assembly comprising at least one camera arranged on a gantry, the at least one camera having a horizontal field of view having a size covering at least the width of the vehicle and allowing a video stream of objects present on the vehicle to be collected when the vehicle passes under the gantry;

[0026] b. Providing a processor, for example embedded in the camera, or even integrated into a computing facility located in an auxiliary location near the gantry, or even remotely (cloud computing), which processor is configured to:

[0027] i. Processing the video transmitted by the one or more cameras to separate the video sequence containing the vehicle of interest passing through and determine the number of objects present on the vehicle and the associated uncertainty;

[0028] ii. comparing the number of objects thus determined with a target value, for example a target value corresponding to an expected number of objects, for example an expected number of objects specified in an information system, for example a delivery note for the delivery of the objects to the site in question;

[0029] c. Advantageously, display devices are provided that interact with the processor and that allow the determined number of objects, the associated uncertainty and the target value to be published simultaneously on these display devices;

[0030] d. If the difference between the determined number of objects and the target value is greater than a given set point, the processor commands the execution of one or more actions, such as the transmission and display of an abnormal alarm, or even a manual recount of the load in question by field personnel;

[0031] The vehicles of interest whose loads need to be counted are equipped with a unique pattern recognizable by the processor, allowing the vehicles of interest to be distinguished from other elements passing in the vicinity of the gantry (hence through the field of view and analysis of the camera), whether these elements are of the same nature or of different nature (such as cars, trucks, tankers, bicycles, pedestrians, forklifts, etc.), said unique pattern being composed, for example, of one or more ArUco markers placed on each vehicle of interest, or even of any other recognizable unique pattern such as a QR code.

[0032] It should therefore be understood that the significance of the invention is that it is intended to enable the distinction of what it considers to be “vehicles of interest”, i.e. vehicles whose loads need to be counted, so that these vehicles of interest can be visually distinguished from other elements passing through the area, whether these elements are of the same nature or of different nature.

[0033] Furthermore, in this example, according to the invention, it is necessary to be able to distinguish vehicle A, which transports gas cylinders, from another vehicle B, which also transports gas cylinders, although the two vehicles of interest are of the same nature (for example, because certain logistic flows must be excluded from the count for given reasons X or Y). This distinction cannot be made naturally, either on the basis of the appearance of these vehicles or on the basis of the content of their loads. However, the number of objects counted must be compared with the valid number contained in an information system (for example a delivery note).

[0034] In other words, it will be appreciated that the method of the present invention allows:

[0035] -Select the vehicle to focus on in the video stream;

[0036] - and discontinuously count the objects present on these vehicles of interest;

[0037] Keep in mind that the vehicle’s load is partially or fully visible via the video stream at any given point in time;

[0038] - It is also possible to perform a panoramic reconstruction of the vehicle's load, which can be likened to a top-view reconstruction based on video images, making it possible to visualize the load of objects, such as gas cylinders, and their distribution on the vehicle, such as groups of gas cylinders of different sizes and colors, or equipped with different forms of gas cylinder caps, etc.

[0039] The invention also relates to a device for discontinuously counting objects present on a moving vehicle, said movement being in a continuous stream along a straight or substantially straight path, said vehicle being considered as a vehicle of interest capable of being identified relative to other vehicles of the same or different nature, for example for counting gas cylinders loaded on a moving vehicle, for example vehicles entering and / or leaving an industrial site, said device comprising the following elements:

[0040] a) at least one gantry equipped with at least one camera having a horizontal field of view having a size covering at least the width of the vehicle and allowing to collect a video stream of objects present on the vehicle as the vehicle passes under the gantry;

[0041] b) a processor, for example embedded in the camera, or even integrated into a computing facility located in an auxiliary location near the gantry, or even remotely (cloud computing), configured to:

[0042] i) processing the video transmitted by the one or more cameras to separate the video sequence containing the passing of the vehicle of interest and determining the number of objects present on the vehicle and the associated uncertainty;

[0043] ii) comparing the determined number of objects with a target value, for example a target value corresponding to an expected number of objects, for example an expected number of objects specified in an information system, for example a delivery note for the delivery of the objects to the site in question;

[0044] c) advantageously comprises display means interacting with the processor, these means being able to publish the number of objects determined, the associated uncertainty and the target value;

[0045] The vehicles of interest whose loads need to be counted are equipped with a unique pattern, for example consisting of one or more ArUco marks placed on each vehicle of interest, or even consisting of any other unique pattern placed on each vehicle of interest, such as one or more QR codes, and the processor is able to recognize the unique pattern and thus distinguish the vehicle of interest from other elements passing near the gantry, whether these elements are of the same nature or of different nature.

[0046] The present invention may implement one or more of the following embodiments:

[0047] - the camera is compatible with an externally used RGB camera and is capable of collecting high-resolution video streams;

[0048] - the processor is a CPU / GPU;

[0049] - The processor uses a software composed of two modules: the first module is based on artificial intelligence, of the type of deep learning or deep neural networks, and the second module is a counting algorithm based on image processing;

[0050] - the device is located at the entrance and / or exit of the site in question so as to allow the handling of vehicles entering and / or leaving the site without these vehicles having to cross the path under the same gantry;

[0051] - The device is located at a single location, either the entrance or exit of the site in question, and allows the handling of vehicles entering and exiting through that single location, two vehicles cannot simultaneously cross the path under the gantry equipped at that location.

[0052] However, other means may be used to perform such functions without departing from the scope of the invention, for example:

[0053] - Infrared camera, especially useful for nighttime acquisitions: the video will be in grayscale instead of color, so this mode prevents obtaining information about the color of the gas cylinder, which can provide useful information about the nature of the gas. For nighttime acquisitions or acquisitions in darkness, additional lights can be used.

[0054] - A processor configured with "tracking" software: allows a unique tracking number or "identifier" to be attributed to each object in the video (so as not to count the same object twice), and finally the number of identifiers is counted to obtain the number of objects (e.g. gas cylinders), which is usually used when video streams are involved.

[0055] The concept of generally straight-line direction or "one-way" will be clarified below.

[0056] According to the invention, the situation in which the vehicle travels along a straight or substantially straight route is preferred, which can be achieved, for example, by using a narrow road upstream of the gantry, or by installing boundaries such as bollards, continuous guide line markings, etc.

[0057] Using such a configuration a preferred straight path can be obtained.

[0058] In this context, if a given reference axis is considered (eg a north-south axis), the route may be at any angle to that reference axis, as long as the adopted "off" axis remains substantially straight.

[0059] Then, for example, the camera can be positioned so that the apparent path of the truck in the video is vertical, but this is just a non-limiting example. If the path of the truck appears to be tilted in the video, then according to the invention it is recommended to rotate the camera to obtain the desired result.

[0060] In summary, the route followed by the gantry upstream will form a straight or nearly straight path, and the camera will be positioned in a way that will allow the angle of this path to be set to facilitate the counting method.

[0061] The case of a delivery note for delivering gas cylinders to an industrial site (eg, a gas cylinder filling site) will be considered below.

[0062] Typically, the following scenarios are encountered, but they merely illustrate the wide variety of scenarios that may arise:

[0063] During the delivery (and collection) of cylinders to manufacturers, laboratories, universities, etc., the driver usually reports manually on the delivery note any discrepancies observed during the delivery (missing cylinders, cylinders already collected in a previous round, etc.). It is therefore contemplated that, according to the invention, the processor calculates the number of cylinders based on the video and the associated uncertainties, compares it with the indication given on the delivery note, and displays the result. Since the driver knows the expected number of cylinders (from the delivery note), he or she is able to check this display and possibly initiate a manual recount.

[0064] ● In another conceivable scenario, the driver reports any discrepancies for the round on a mobile application that is synchronized with the resource and cylinder delivery plan. Upon return to the plant, the processor calculates the number of cylinders based on the video and the associated uncertainty, retrieves the expected number of cylinders from the plan, verifies that this value is indeed within the prediction interval, and then displays the result. The processor can then trigger / optionally display an alarm, for example, to initiate a manual recount.

[0065] One example of a process for processing an image received by a processor will now be described.

[0066] a) Starting from a continuous (video) stream, separating out sequences of interest corresponding to the passage of vehicles: for example using a binary classifier, such as a neural network, i.e. a mathematical model trained to recognize the presence or absence of vehicles of interest in an image, this model being associated for example with a detector of ArUco markers that are placed on each vehicle of interest in order to be able to identify these vehicles of interest and distinguish them from other vehicles that are therefore considered irrelevant.

[0067] b) Detecting objects of interest on the vehicle using an artificial neural network, ie, a mathematical model trained to identify objects of interest in each image.

[0068] c) Estimate the pixel-wise motion rate of the vehicle by using the detection coordinates (obtained by inference) in consecutive images.

[0069] d) The purpose of the processing is to binarize the image by setting the pixels in the following areas to 1 (or other values ​​that allow distinguishing the object categories):

[0070] i) the region is defined, for example, by the detection coordinates having a score above a given threshold,

[0071] ii) the area is transformed into a circle whose diameter is smaller than a given factor and smaller than the minimum between the detection width and height,

[0072] And set the rest to 0.

[0073] e) Estimate the movement of the vehicle in pixels, taking into account the results of c). Since the movement between the two images can be assumed to be linear, the rotation of the images allows to simulate linear vertical movement.

[0074] f) defines N rows:

[0075] i) Perpendicular to the movement of the vehicle

[0076] ii) Placed at various horizontal positions in the image

[0077] iii) Its thickness is equal to the vehicle movement in pixels calculated in section e). The strips extracted for each corresponding image are combined vertically to reconstruct the "count panorama". The same process is also applied to the background image to reconstruct the "vehicle panorama".

[0078] C iterations of morphological closing are applied to the N count panoramas obtained in f)iii).

[0079] g) Count the number of elements in each of the N puzzles / mosaics, optionally filter out elements of inconsistent size, and then derive the final count and uncertainty by computing the weighted average (the center row has more weight than the rows at the ends) and standard deviation, respectively.

[0080] Attached Figure 1 One example of an apparatus suitable for implementing the invention is shown, here employing a processor integrated into a computing facility located in an auxiliary location near the gantry.

[0081] Figure 1 The naming of the components presented in is as follows:

[0082] -1: Truck transporting the cylinders to be counted (4) (shown in the background and drawn with a light line)

[0083] -2: Display device

[0084] -3: Computer

[0085] -5: Entrance Camera

[0086] -6: Exit Camera

[0087] -7: (Information flow from camera to computer, represented by bold line)

[0088] Thus, the examples show an arrangement comprising two cameras on the gantry; however, as mentioned above, a single camera is sufficient in the context of the present invention.

Claims

1. A method for discontinuously counting objects (4) present on a moving vehicle (1), said movement being in a continuous stream along a straight or substantially straight path, said vehicle being considered as a vehicle of interest capable of being identified relative to other vehicles of the same or different nature, for example for counting gas cylinders loaded on a moving vehicle (e.g. a vehicle entering and / or leaving an industrial site), wherein: a. providing at least one assembly comprising at least one camera (5, 6) arranged on a gantry, the at least one camera having a horizontal field of view, the size of which covers at least the width of the vehicle and allows collecting a video stream of objects present on the vehicle when the vehicle passes under the gantry; b. Providing a processor (3), for example embedded in the camera, or even integrated into a computing facility located in an auxiliary location near the gantry, or even remotely (cloud computing), and using the processor to implement the following measures: i. Processing the video transmitted by the one or more cameras to separate the video sequence containing the vehicle of interest and determine the number of objects present on the vehicle and the associated uncertainty; ii. comparing the number of objects thus determined with a target value, for example a target value corresponding to an expected number of objects, for example an expected number of objects specified in an information system, for example a delivery note for the delivery of the objects to the site in question; c. Advantageously, display means are provided that interact with said processor, which allow the determined number of objects, the associated uncertainty and the target value to be published simultaneously on these display means; d. If the difference between the determined number of objects and the target value is greater than a given set point, the processor commands the execution of one or more actions, such as the transmission and display of an abnormal alarm, or even a manual recounting of the load in question by field personnel; The vehicles of interest whose loads it is desired to count are equipped with a unique pattern recognizable by the processor, and the processing allows distinguishing the vehicles of interest from other elements passing in the vicinity of the gantry, whether these elements are of the same nature or of different nature, the unique pattern consisting, for example, of one or more ArUco markers placed on each vehicle of interest, or even of any other recognizable unique pattern (such as a QR code), the processor being equipped with a mathematical model trained to detect and recognize the presence of such patterns and therefore of the vehicles of interest in the video sequence and therefore distinguishing them from other vehicles that would thereby be considered irrelevant.

2. The method according to claim 1, characterized in that The determined number of objects present in each vehicle is differentiated in order to classify the counted number of objects into determined object subcategories, such as gas cylinder colours, or even gas cylinders with different types of caps or the like.

3. The method according to claim 1 or 2, characterized in that: The at least one assembly is positioned at the entrance and / or exit of the site in question so as to allow vehicles entering and / or leaving the site to be handled without the need for said vehicles to pass through a path underneath the same gantry.

4. The method according to claim 1 or 2, characterized in that: The at least one assembly is positioned at a single location, either an entrance or an exit to the site in question, and allows the handling of vehicles entering and leaving through this single location, without two vehicles being able to simultaneously cross the path underneath a gantry equipped at this location.

5. The method according to any one of the preceding claims, characterized in that The at least one camera is an RGB camera compatible with external use and capable of collecting a high-resolution video stream.

6. The method according to any one of the preceding claims, characterized in that The processor is a CPU / GPU processor.

7. The method according to claim 6, characterized in that The processor uses software consisting of two modules: the first is based on artificial intelligence, of the type known as deep learning or deep neural networks, and the second is a counting algorithm based on image processing.

8. The method according to any one of the preceding claims, characterized in that During the processing a panoramic reconstruction of the load of the vehicle of interest is performed.

9. A device for discontinuously counting objects (4) present on a moving vehicle (1), said movement being in a continuous stream along a straight or substantially straight path, said vehicle being considered as a vehicle of interest capable of being identified relative to other vehicles of the same or different nature, for example for counting gas cylinders loaded on a moving vehicle (e.g. a vehicle entering and / or leaving an industrial site), said device comprising the following elements: a) at least one gantry equipped with at least one camera (5, 6) having a horizontal field of view the size of which covers at least the width of the vehicle and allows collecting a video stream of objects present on the vehicle as it passes under the gantry; b) a processor, for example embedded in the camera, or even integrated into a computing facility located in an auxiliary location near the gantry, or even remotely (cloud computing), configured to: i) processing the video transmitted by the one or more cameras to separate the video sequence containing the passage of the vehicle of interest and determining the number of objects present on the vehicle and the associated uncertainty; ii) comparing the determined number of objects with a target value, for example a target value corresponding to an expected number of objects, for example an expected number of objects specified in an information system, for example a delivery note for the delivery of the objects to the site in question; c) advantageously comprising display means interacting with said processor, these means being able to publish the number of objects determined, the associated uncertainty and the target value; The vehicles of interest whose loads it is desired to count are equipped with a unique pattern, consisting, for example, of one or more ArUco marks placed on each vehicle of interest, or even of any other unique pattern placed on each vehicle of interest, such as one or more QR codes, and the processor is able to recognize the unique pattern and thereby distinguish the vehicle of interest from other elements passing near the gantry, whether these elements are of the same or different nature.

Images