Method and system for assessing distance of object viewed by camera

Abstract

The subject matter of the invention is a method and system for assessing the distance of an object viewed by a camera. The invention relates to a system for measuring the distance of an object (O1-O3). The system comprises a single camera (2) and a processing device (3) connected to the camera (2). The camera (2) captures an image received by the processing device (3). The processing device (3) runs an image recognition algorithm which recognizes, for each object recognized in the acquired image, the type to which the object belongs and defines a bounding box surrounding the object. In addition, the processing device (3) determines a reference dimension of the bounding box. In particular, the portable device corrects a reference size of a bounding box of the identified object by applying a correction criterion selected according to a value of at least one pre-stored parameter associated with a type of the identified object and at least one feature of the bounding box of the identified object. Finally, the processing device (3) calculates the distance to the object as a function of the reference dimension thus corrected and at least one comparison dimension stored in the system.

Description

Technical Field

[0001] The present invention relates to the field of image processing for the purpose of estimating the distance to one or more viewed objects. In particular, the present invention is applied in portable systems, especially personal systems (e.g., wearable devices). For example, the present invention is used in collision avoidance systems, and in particular in systems designed to warn users, such as skiers, cyclists, motorcyclists, etc., of an impending collision with an object outside the user's field of view. Background Art

[0002] Object detection around an item of interest (especially in motion) is a feature used in many application areas today, be it independently used in traffic monitoring systems (e.g. to detect violations), autonomous and / or assisted mobility systems, augmented reality systems, etc.

[0003] For example, in the sports field, collision avoidance systems are becoming increasingly popular, particularly in the automotive industry, where cars are equipped with a large number of sensors capable of periodically detecting objects in the surrounding environment. In this field, systems based on the analysis of objects within the frames captured by cameras, data provided by radar, lidar and / or ultrasonic sensors are known.

[0004] Information about detected objects is typically provided to a tracking system that assigns a unique ID to each object detected at a given time and is able to track each of these objects in subsequent frames in order to determine if objects are approaching and assist the driver in avoiding collisions with them.

[0005] For example, US 2009 / 0195371 and EP 3279830 describe collision avoidance systems that can use a tracker algorithm to determine the distance to an object detected in an image. Compared to conventional object tracking algorithms, the systems use simplified pre-processing expedients to facilitate the initiation of safety measures to reduce the risk of collisions in the automotive industry.

[0006] However, known systems include sensors that are expensive, bulky, energy-consuming, and / or unsuitable for personal use due to the radiation they emit; in particular, these systems are inaccurate in the case of non-horizontal cameras and / or objects that are not considered upright or in objects whose surfaces are not perpendicular to the camera due to yaw (i.e., horizontal rotation of the camera and / or rotation of the object about its vertical axis).

[0007] These factors hinder their application in various fields, such as motorcycling, cycling, water sports (e.g. surfing) or skiing, where the risk of collision is anyway high. In particular, in skiing, cycling, water-based activities and general personal use, radars and lidars that would otherwise provide information even about the distance to objects are not available due to their weight, energy consumption and health-related issues (because of the intensity of the emitted electromagnetic radiation). In these fields, estimating distances by parallax analysis between images obtained from multiple cameras is also impractical due to the minimum spacing required between the cameras and the relative bulkiness. Therefore, in these cases, it is possible to develop a system comprising a single camera, downstream of which (independently as in US2020 / 394435) a machine vision-based object recognition algorithm is executed to recognize the presence of certain classes of objects in the image and to delimit them using bounding boxes whose sides are parallel to the sides of the imaged image and whose two dimensions are analyzed to estimate the distance to each of these objects.

[0008] In this case, the only data available for each frame is the recognition confidence percentage, two bounding box sizes, and the bounding box position for each object identified in the frame.

[0009] However, unlike in the automotive sector, where changes in the car's tilt are minimal, in skiing, cycling, and motorcycling, cameras are subject to extensive and unpredictable changes in tilt and orientation during use, making the point of view and field of view of the acquired images highly variable. Camera tilt (especially yaw) introduces errors into computer vision-based distance measurement systems because it causes significant changes in the size of the bounding box that delimits a given identified object and, therefore, errors in the distance estimate. Similar errors can occur with objects that are not necessarily upright, as they do for skiers, cyclists, and motorcyclists when they are turning. Furthermore, the tilt of an object causes a change in the size of the bounding box, but does not provide any information about the object's tilt within the bounding box. Furthermore, for the same actual distance, changing the size of the bounding box results in corresponding errors in the calculation of the object's distance. Another source of error is the rotation of the object about its vertical axis relative to the camera surface, which again results in a change in the size of the bounding box at the same distance and, therefore, errors in the distance calculation.

[0010] Other known systems, such as US2019 / 355140, describe systems based on stereo vision and do not suffer from these errors, but as explained, they are not suitable for personal use because there is not enough space for a second camera at an appropriate distance from the first camera. Monocular systems such as US2020 / 394435 are inappropriate because they are susceptible to the errors mentioned above. Finally, systems such as US2019 / 318481 describe how to correct errors caused by the vertical tilt (i.e., pitch) of cameras in the automotive field, but at least require a second means for estimating the distance to at least one of the imaged objects, in specific cases by means of a lidar, radar, sonar, or unspecified machine learning algorithm to determine the need for pitch correction and apply it. However, as explained earlier, in the case of portable personal use, it is not possible to be equipped with tools such as lidar or similar detectors, and using machine learning to estimate the distance to an object (assuming it is feasible in practice) still involves a large computational load that imposes demanding requirements on the hardware required for such applications. Additionally, US2019 / 318481 corrects only for changes in the camera's vertical axis, but does not allow correction for errors due to rotation of the camera (which is very common in non-automotive personal use), nor does it allow correction for errors due to changes in tilt or rotation of the observed object (which is also very common in the case of, for example, skiers, cyclists, and motorcyclists).

[0011] In the aforementioned fields, there is a need for a system capable of detecting objects in the surrounding environment and estimating their distances by correcting the aforementioned errors in a fast, low-energy manner, and using a single camera as a sensor, in particular for providing a portable personal collision avoidance system. For example, skiers are frequently run over by other skiers from directions outside the skier's field of view (e.g., from behind), sometimes with fatal consequences. Similarly, cyclists or motorcyclists are frequently struck or run over by cars or other vehicles approaching from a direction outside the driver's field of view.

[0012] Therefore, it is felt that there is a need for a system that can determine the distance to one or more objects with reasonable reliability, which system is lightweight, does not use bulky and energy-intensive hardware such as radar or lidar, does not require two or more cameras or require special camera orientations, can determine whether correction is needed, and correct for errors introduced by rotation on or about vertical line of the object and / or camera rotation in case of any error (does not require high computing power), does not require high power consumption, and does not emit harmful radiation to a user carrying such hardware or located near it. Summary of the Invention

[0013] The object of the present invention is to overcome the disadvantages of the prior art.

[0014] In particular, the object of the present invention is to provide systems and methods that can robustly and reliably estimate the distance between an object framed by a single camera and the camera itself without the aid of computationally expensive machine learning-based distance measurement systems or other types of sensors (e.g., radar, lidar, sonar, etc.) or additional cameras.

[0015] The system and method according to the present invention determines that corrections are needed and corrects errors caused by changes in the system camera settings and / or the tilt / rotation of the object being imaged relative to its vertical / horizontal lines. Advantageously, each necessary correction is applied, preferably by a machine vision algorithm, using only data provided by the system for the bounding box of the object imaged in the image.

[0016] Furthermore, it is an object of the present invention to provide a system and method suitable for determining whether data provided by a machine vision system is unreliable.

[0017] An additional object of the present invention is to provide systems and methods that are not harmful to health, in particular those that are not based on potentially harmful electromagnetic radiation, such as radar waves or lasers.

[0018] Furthermore, it is an object of the present invention to provide a system that is portable, preferably wearable, lightweight, unobtrusive and requires little electrical energy for operation.

[0019] These and further objects of the invention will become more apparent from the following description and from the appended claims, which form an integral part of this description.

[0020] According to a first aspect, the present invention relates to a system for measuring the distance of an object. The system comprises a single camera and a processing device connected to the camera. The camera acquires an image, which is received by the processing device. The processing device pre-stores, for each identifiable object type from a plurality of types, at least one parameter among an expected shape factor between the height and width of a bounding box of at least one reference object, expected depth data (preferably assigned as a function of the change in the measured shape factor of the reference object as the rotation angle of the front face of the reference object relative to the camera changes), and expected verticality data (preferably determined based on at least one reference object of the object type, the positioning of the camera, and / or the presence / absence of horizon correction of the image acquired by the camera).

[0021] The processing device executes an image recognition algorithm that identifies, for each object identified in the acquired image, the type to which the object belongs among a plurality of identifiable object types (e.g., included in a list of possible objects). Typically, the image recognition algorithm is based on computer vision, and the plurality of identifiable object types includes object types or object-types used during the training of the algorithm itself.

[0022] For example, image processing can be performed by an object recognition algorithm based on computer vision. Preferably, the algorithm is a convolutional neural network or CNN (convolutional neural network) independent of the Fast R-CNN or Faster R-CNN type. In a non-limiting embodiment, the computer vision algorithm is trained to recognize images using a general dataset such as COCO 2017 that includes objects such as people (skiers), plants and / or vehicles. More generally, the computer vision algorithm is trained using at least one dataset specific to the domain in which the system is used.

[0023] For each identified object, the system defines a bounding box that encloses the object. Typically, the bounding box is a rectangle whose horizontal sides are parallel to the top and bottom edges of the image fed to the processor and whose dimensions are such that it encloses the horizontal and vertical ends of the identified object.

[0024] The system comprises calculating a reference dimension of a bounding box, such as its diagonal, its area, or the number of pixels it contains. The system corrects the reference dimension of the bounding box of the identified object by applying a correction criterion. Advantageously, the correction criterion is selected based on the value of at least one pre-stored parameter associated with the type of the identified object and at least one characteristic of the bounding box of the identified object (e.g., shape factor, area, etc.). Finally, the system calculates the distance to the identified object based on the thus corrected reference dimension and at least one comparison dimension stored in the system, wherein the at least one comparison dimension is associated with a corresponding bounding box enclosing a known object of the same type located at a known distance.

[0025] Thanks to the system of the invention, it is possible to obtain an estimate of the distance of an identified object without having recourse to a stereo system. Furthermore, it is possible to obtain such a distance estimate substantially in real time even in systems comprising limited hardware resources.

[0026] Furthermore, the system provides reliable estimates by correcting for any estimation errors. In practice, image-based estimates of object distances can be distorted by misalignment and / or rotation of the camera's vertical relative to the object's vertical (hereinafter referred to as "tilt error") and / or by rotation of the object itself (about its own vertical) relative to the camera (particularly relative to the camera's framing in the horizontal direction) (hereinafter referred to as "rotation error"). For example, the bounding box of a skier tilted relative to the camera is much wider and only slightly lower than the bounding box of a skier standing upright relative to the camera, resulting in a larger diagonal than would be the case if the skier at the same distance were not tilted, and thus subject to tilt error that leads to an underestimation of the distance. Conversely, the bounding box of a very wide and shallow object whose primary face is not directly facing the camera (i.e., the object is rotated about its vertical relative to a position where its primary face is facing the camera) results in a narrower bounding box than expected, and therefore has a shorter diagonal. Consequently, in this case, rotation error can lead to an overestimation of the actual distance. Conversely, in cases where the object is very deep relative to its height, such as a bus, the rotation error introduced by the rotation of the object relative to the camera can lead to an underestimation of the distance.

[0027] These errors are very common in real-life conditions. For example, tilt errors can be caused by the tilt of the camera itself compared to the horizon, which can be almost certain if the camera is worn by the user while on the move and without a system that rotates the image so that the horizon is always parallel to the upper and lower edges of the image. Or they can be caused by the tilt of the identified object relative to the horizon for objects such as skiers, cyclists, motorcyclists, bicycles and motorcycles that may be tilted relative to the horizon and / or the camera (even if the latter is equipped with horizon correction) (unlike cars, trees, etc., which generally do not tilt relative to the horizon). Conversely, rotation errors occur in all cases where the main face of the object is not facing the camera exactly (that is, substantially parallel to the plane of the acquired image).

[0028] The system of the present invention allows these errors to be corrected automatically and substantially in real time by identifying whether and what correction criteria to apply, based on pre-stored information about the type of object under consideration and information about the bounding box of the object under consideration. In other words, compared to known systems and using only one camera as a detector, the system provides a fine estimate of the distance to one or more objects identified by the camera at a significantly lower computational cost (i.e., without using other sensors (e.g., sonar, lidar, radar) and / or methods (e.g., machine learning algorithms for coarse distance estimation, assuming they are achievable)).

[0029] The above parameters can be calculated from images acquired using a camera having characteristic parameters different from those of the cameras included in the system. In this case, any differences between the two (or more) cameras (such as the ratio between vertical and horizontal resolution and other technical parameters) are compensated so that the stored parameters are equivalent to the parameters calculated from images acquired using a camera having characteristic parameters equivalent to those of the system cameras.

[0030] Optionally, for each estimated distance value, the system indicates a corresponding degree of reliability of the estimate.

[0031] In an embodiment, at least one pre-stored parameter for each object type is determined by identifying a reference object in a reference image and creating a corresponding reference bounding box. Preferably, the vertical axis of the reference object is parallel to the vertical axis of the camera, and the front face of the reference object is perpendicular to the image plane.

[0032] In an embodiment, the expected depth data for each type of object having a square outline is logical data having a true value if:

[0033] cos(Am)+sin(Am) / Rp<1

[0034] where Am is the maximum track angle relative to the camera for an object type defined in more detail below and Rp is the average width to depth ratio for the object type, and otherwise the expected depth data exhibits spurious values.

[0035] In one embodiment, the expected verticality data (also indicated as a parameter of possible non-perpendicularity) depends on whether the vertical axis of the object type is clearly aligned with the vertical axes of the vertical edges of the image. The parameter of possible non-perpendicularity is only true for objects that are generally "not tilted" (independently of cars, buses) and are captured by a camera with a horizontal position or equipped with horizon correction.

[0036] Preferably, the processing device pre-stores all parameters for each type of object, namely the expected shape factor, the expected depth data, and the expected verticality data. Furthermore, it selects which correction to apply to the reference dimension based on the values ​​of these pre-stored parameters associated with the type of recognized object and the shape factor of the bounding box of the recognized object.

[0037] In one embodiment, the processing device corrects the reference size by applying a tilt error correction criterion, by computing a corrected bounding box having an expected shape factor and having corners that lie on the sides of the bounding box of the identified object.

[0038] Preferably, but not exclusively, the processing means identifies the need for a single tilt error correction when:

[0039] Ra > 1, Pa = false, Va = false and Rm < Ra, or

[0040] Ra < 1, Pa = true and Rm > Ra

[0041] where Ra is the expected shape factor of the recognized object type, Pa is the expected depth value, Va is the expected verticality value and Rm is the measured shape factor.

[0042] The correction of the tilt error is performed only on the basis of the measured and expected shape ratios. Additionally, the correct shape factor can be determined by geometric / trigonometric techniques, preferably by calculating a corrected bounding box having the expected shape factor and an angle located on the side of the bounding box of the recognized object. Thus, the system according to the present invention allows for the effective correction of any tilt error through a series of operations with low computational complexity.

[0043] Preferably, the system includes an algorithm or other horizon correction function. Horizon correction allows for the elimination of tilt error occurrences due to the non-horizontal deflection orientation of the camera. Due to the use of horizon correction, tilt error compensation occurs only for objects that are not necessarily in a vertical position by nature (such as skiers, cyclists, etc.). In this case, only this last group of object types has the parameter Pa = false. Otherwise, if horizon correction is not available, then all possible object types have the parameter Pa = false.

[0044] In one embodiment, the portable device corrects the reference dimension by applying a rotation error correction criterion and by calculating the correct width of the bounding box as the ratio between the height of the bounding box enclosing the object and the expected shape relationship of the recognized object type.

[0045] Preferably, but not exclusively, the processing device identifies the need for a single rotation error correction when:

[0046] Ra > 1, Pa = false, Va = false and Rm > Ra,

[0047] Ra > 1, Pa = false, Va = true and Rm > Ra,

[0048] Ra > 1, Pa = true, Va = true and Rm < Ra,

[0049] Ra < 1, Pa = false, Va = true and Rm > Ra,

[0050] Ra < 1, Pa = true, and Rm < Ra,

[0051] Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, where 0 ≤ x ≤ 20, Pa = false, and Rm > Ra, or

[0052] Ra≈1, preferably 1 - x% ≤ Ra ≤ 1 + x%, where 0 ≤ x ≤ 20, Pa = true, and Rm < Ra, where Ra is the expected shape factor of the recognized object type, Pa is the expected depth data, Va is the expected verticality data and Rm is the measured shape factor.

[0053] Starting from the height data of the bounding box of the recognized object and the expected shape factor (i.e., pre - stored for that type of object), rotation error correction is applied. Similar to tilt error correction, the rotation error is also compensated by an operation with extremely low computational cost.

[0054] In addition, the applicant notes that there are cases where it is certain that at least one of tilt and rotation correction should be applied but it is impossible to establish which of the two corrections to apply and / or cases where the vertical axis of the imaged object is tilted relative to the vertical edge of the image and the object itself is also rotated. For simplicity, such cases are hereinafter referred to as "compound errors". In these cases, the corrected reference size of the bounding box is calculated as the average of a first reference size calculated by applying tilt error correction to the bounding box enclosing the recognized object and a second reference size calculated by applying rotation error correction to the bounding box enclosing the recognized object. Preferably, when applying compound error correction, the system signals that the data obtained by means of this correction has lower reliability compared to when only applying a tilt error correction or only applying a rotation error correction.

[0055] In other words, according to this embodiment, the processing device corrects the reference size by applying a compound error correction criterion and by calculating the corrected reference size of the bounding box as the average of a first reference size calculated by applying tilt error correction to the bounding box enclosing the recognized object and a second reference size calculated by applying rotation error correction to the bounding box enclosing the recognized object.

[0056] Preferably, but not limited to, the processing device identifies the need for compound correction when:

[0057] Ra > 1, Pa = true, Va = false and Rm < Ra, or

[0058] Ra < 1, Pa = false, Va = false and Rm > Ra,

[0059] where Ra is the expected shape factor of the recognized object type, Pa is the expected depth data, and Va is the expected verticality data.

[0060] The correction of the composite error is obtained by averaging the tilt and rotation corrections described above. Thus, the correction of the composite error can also be achieved with a relatively low computational cost.

[0061] In one embodiment, given the recognized object type and data derived from the bounding box of the recognized object, the system reports cases where the information determined by the recognition algorithm is considered unreliable or when, for example, it should not actually occur. Preferably, but not limited to, an alarm signal is generated when:

[0062] Ra > 1, Pa = false, Va = true and Rm < Ra,

[0063] Ra > 1, Pa = true and Rm > Ra,

[0064] Ra < 1, Pa = false and Rm < Ra,

[0065] Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, con 0 ≤ x ≤ 20, Pa = false, and Rm < Ra, or

[0066] Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, con 0 ≤ x ≤ 20, Pa = true, and Rm > Ra.

[0067] Preferably, when the system identifies one of these cases, the data related to the bounding box of the recognized object (i.e., one or more features) is discarded as unreliable.

[0068] Finally, in one embodiment, no correction is applied when:

[0069] Rm = Ra ± x%, con 0 ≤ x ≤ 20, where Rm is the shape factor calculated by the processing device and Ra is the expected shape factor of the recognized object type. In other words, minor differences between Rm and Ra are not considered.

[0070] By identifying the need for the required correction and correction type based on the considered reference dimensions and pre - stored parameters of each possible type of recognized object, it is possible to calculate a fine estimate quickly and at a low computational cost, allowing the system to operate properly in real - time even when the system includes limited hardware resources (such as portable and / or consumer devices).

[0071] In one embodiment, each constant value stored by the processing device is a function of a comparison dimension associated with the comparison of the bounding box of an object of the same type as the recognized object and located at a known distance, where the size of the comparison dimension of the comparison bounding box is the same as the size of the reference dimension associated with the bounding box of the enclosing object.

[0072] The system of the present invention makes it possible to obtain an estimate of the distance of an identified object without resorting to a stereo system or other sensors or machine learning systems. Furthermore, it is possible to obtain such an estimate of the distance substantially in real time even in systems comprising limited hardware resources.

[0073] In one embodiment, the comparative size is calculated starting from an image depicting a known object with its vertical axis parallel to the vertical axis of the image and with the front face of the object parallel to the image plane.

[0074] In one embodiment, the reference size and the comparison size each correspond to a diagonal of a bounding box surrounding the identified object or a known object of the same type at a known distance from the system.

[0075] Due to these characteristics, the system comprises the information necessary to quickly and accurately determine the distance of the identified objects, while also including requirements regarding the memory space necessary for the system.

[0076] In one embodiment, the distance between the object and the camera is calculated with the help of the following relationship:

[0077]

[0078] Where D is the calculated diagonal of the recognized object's bounding box, and Kc is a constant value associated with the recognized object pre-stored on the device.

[0079] Preferably, the constant value Kc is determined relative to each identifiable object type by drawing one or more reference objects for each object type in a vertical position with its own front face (relative to the typical movement direction of the object itself) directly facing the camera and with the vertical face of the object parallel to the camera's viewing axis.

[0080] In one embodiment, the constant value is a function of a field of view associated with the camera and a unit of measurement used to measure the size of the bounding box.

[0081] Preferably, the comparative size is calculated based on a sample image acquired by a camera (preferably the same or similar to the system camera) with known characteristic parameters (field of view and / or number of pixels). In other embodiments, the camera used to acquire the comparative image is different from the system camera. In this case, one or more characteristic parameters of the camera that acquired the sample image (e.g., field of view, horizontal and vertical resolution, shape factor, etc.) are taken into account to compensate for the differences between the cameras, thereby avoiding the introduction of errors in the calculation of the comparative size. As an alternative to the characteristic parameters of two or more cameras, the compensation for the different cameras is based on one or more conversion parameters between the characteristic parameters of the system camera and one or more of the cameras used to acquire the reference size.

[0082] In an embodiment, the processing means is adapted to calculate the calculated distance reliabilities, wherein each reliability is calculated based on at least one parameter associated with the type of the corresponding identified object, and preferably each reliability is calculated based on the difference between an expected shape factor of an object of the identified type and a measured shape factor of the identified object.

[0083] Various aspects of the invention relate to a method for estimating a distance between an object and a system comprising a single camera and a processing device connected to the camera.

[0084] The method includes acquiring at least one image using a camera. In addition, the method includes pre-storing, by a processing device, at least one parameter of the following for each identifiable object type:

[0085] an expected shape factor between a height and a width of a bounding box of at least one reference object, expected depth data (determined as a function of a change in the measured shape factor of the reference object as a rotation angle of the front face of the reference object relative to the camera changes), and

[0086] Expected verticality data (determined based on object type, camera positioning, and / or presence / absence of horizon correction of images acquired by the camera).

[0087] Furthermore, the method provides for an electronic processor to perform the steps of receiving the acquired image and executing an image recognition algorithm, the image recognition algorithm for each object identified in the acquired image:

[0088] identifying the type of object to which the identified object belongs among a plurality of identifiable object types, and

[0089] define a bounding box around the identified object, and

[0090] Determines the reference dimensions of the bounding box.

[0091] Next, the computer performs the steps of determining a reference size of the bounding box and correcting the reference size of the bounding box of the identified object by applying a correction criterion, wherein the correction criterion is selected based on a value of at least one pre-stored parameter associated with the type of the identified object and at least one characteristic of the bounding box of the identified object.

[0092] Finally, the electronic processor performs the step of calculating the distance to the identified object as a function of the reference size thus corrected and at least one comparative size stored in the system, the at least one comparative size being associated with a corresponding bounding box surrounding a known object of the same type located at a known distance.

[0093] The method according to the invention allows to obtain, in the embodiment presented mutatis mutandis, the same advantages as described above for the system.

[0094] Further features and objects of the present invention will become apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The present invention will be described herein with reference to some examples provided by way of example and not limitation and shown in the accompanying drawings. These drawings illustrate different aspects and embodiments of the invention and, where appropriate, reference numerals are used to designate reference numerals showing the same structures, components, materials and / or elements in different drawings.

[0096] Figure 1 is a schematic representation of a system according to an embodiment of the present invention used while skiing;

[0097] Figure 2 is a flowchart of an image processing process according to an embodiment of the present invention;

[0098] Figure 3A is Figure 1 Schematic representation of frames acquired by a camera of the system;

[0099] Figure 3B Corresponding to Figure 3A Frame, which shows the Figure 1 The bounding box identified by the system;

[0100] Figure 3C Because Figure 1 A graphical representation of the principles of information calculation for bounding box recognition by the system;

[0101] Figure 4 is a flow chart of a process for calculating the distance to an object according to an embodiment of the present invention;

[0102] Figure 5 is included in Figure 4 Qualitative representation of the tilt error correction step in the process;

[0103] Figure 6 is a flow chart of a process for detecting and reporting an initiating object according to one embodiment of the present invention, and

[0104] Figure 7 is a flow chart of an image processing process according to an alternative embodiment of the present invention. DETAILED DESCRIPTION

[0105] Some preferred embodiments will be described in detail below, although the invention is susceptible to various alternative modifications. In any case, it must be understood that there is no intention to limit the invention to the specific embodiments described, but on the contrary, the invention is intended to cover all modifications and / or uses of the elements or equivalent steps that fall within the scope of the invention as defined in the claims.

[0106] Unless otherwise defined, all technical terms, symbols, and other scientific terms used herein are intended to have the meanings commonly understood by those skilled in the art to which this description belongs. In some cases, terms with commonly understood meanings are defined herein for clarity and / or ease of reference; therefore, the insertion of such definitions in this description should not be construed as representing a substantial difference relative to what is generally understood in the art.

[0107] The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including but not limited to") and are to be considered to also support terms such as "essentially consist of," "essentially consisting of," "to consist of," or "consisting of."

[0108] Unless otherwise stated, the use of "independently," "etc," and "or" means non-exclusive alternatives without limitation. Unless otherwise stated, the use of "including" means "including but not limited to."

[0109] Figure 1 A skier 1 is shown equipped with a camera 2 and a portable device 3 (such as a smartphone) on which a software application 30 (hereinafter app) is installed, capable of communicating with the camera 2 and processing the images acquired by the camera. In an alternative embodiment (not illustrated), the camera 2 and the portable device 3 are included in a single element and / or are integrated into a helmet worn by the skier 1 and any acoustic sound reproducer typically positioned corresponding to the user's ears.

[0110] In a known manner, the camera 2 acquires a series of image frames or frames and transmits them to the portable device 3 .

[0111] In the embodiment under consideration, the camera 2 is mounted so as to capture a portion of the space (i.e., with a field of view FOV) that is not visible to the skier 1. For example, the camera 2 is fixed to the helmet 4 and is aimed so as to frame the space behind the skier 1. Alternatively, the camera 2 may be fixed to another element, for example, a piece of clothing, such as the skier's 1 jacket.

[0112] The portable device 3 then runs an app 30 which implements a collision avoidance method capable of reporting one or more objects (eg other skiers approaching the skier 1 ).

[0113] The anti-collision method according to the embodiment of the present invention includes but is not limited to the following references respectively Figure 2 、 4 and 6 , the process 1000 for identifying an object, the process 2000 for calculating a distance, and the process 3000 for detecting an approaching object.

[0114] The process 1000 is suitable for identifying the image captured by the camera 2. Figure 3A 1 and 2, one or more objects O1-O3 depicted in each frame 10. In particular, the process 1000 robustly identifies the type of object O1-O3 depicted in the frame 10, its position in space relative to the camera 2 that generated the frame 10 and, therefore, the skier 1, and the size of the detected object O1-O3 compared to the field of view of the camera 2.

[0115] Each frame 10 generated by the camera 2 is received by the portable device 3 (step 1001), and preferably, but not exclusively, each frame 10 generated by the camera 2 is associated with a corresponding timestamp ts (step 1002) or with the sequential order in which the frames were acquired. For example, the timestamp ts is included in the metadata of the frame 10 generated by the camera 2 or is determined by the portable device 3 when the frame 10 is received.

[0116] Optionally, and without limitation, any possible orientation of camera 2 in a non-horizontal position, independently of the non-upright position held by skier 1, is considered. In particular, a "horizontal" orientation is an orientation of camera 2 in which a pair of edges 11-14 in frame 10 are parallel to the horizon. Specifically, a check is optionally performed first to determine whether camera 2 has a non-horizontal orientation (decision step 1003). In one embodiment, a "horizon correction" algorithm and / or sensors placed on camera 2 are used to identify the orientation of camera 2. If camera 2 is not in a horizontal position (output branch Y of step 1003), compensation is optionally performed by rotating frame 10 by a certain angle so that the horizon in the frame is parallel to the true horizon (or, similarly, the direction of gravity acceleration is parallel to the direction of true gravity acceleration) (step 1004), wherein this angle is determined by the algorithm and / or sensors placed on camera 2 in a manner that is known per se and not described here for the sake of brevity.

[0117] Without performing compensation for the orientation of camera 2 (output branch N of step 1003) or after such compensation (downstream of step 1004), frame 10 is processed to identify the presence of one or more objects depicted in frame 10 (decision step 1005). Processing of frame 10 is performed, for example, by an object recognition algorithm based on artificial vision (machine vision). Preferably, the object recognition algorithm comprises a convolutional neural network or CNN (convolutional neural network) independent of the Fast R-CNN or Faster R-CNN type. In a non-limiting embodiment, the object recognition algorithm is trained to recognize images using a general dataset, such as COCO 2017, that also includes people (skiers) or vehicles. Additionally or alternatively, the object recognition algorithm is trained using a dataset specific to the system's domain of use. If no object is identified in the frame 10 under consideration (output branch N of step 1005), frame 10 is ignored (step 1006) and a new frame 10 provided by camera 2 is analyzed. In other words, process 1000 is repeated starting from step 1001.

[0118] Differently, when at least one object is identified (output branch Y of step 1005), the processing of frame 10 determines for each identified object O1-O3 (step 1007):

[0119] i) the type of the identified object among those envisioned in the dataset (e.g. person, tree, car, bicycle, etc.),

[0120] ii) As in Figure 3B As schematically shown in FIG, bounding boxes B1-B3, i.e., quadrilaterals of minimum size (wherein the sides are parallel to the edges of the frame 10), contain within them all the points (i.e., pixels) of the frame 10 representing the identified object, and

[0121] iii) preferably, although not necessarily limiting, a confidence value (e.g. a percentage value) of the recognition performed

[0122] iv) X and Y of the pixel of frame 10 corresponding to a predetermined corner (e.g., the upper left corner) of the bounding box

[0123] Y coordinate, where the X coordinate indicates the position of the pixel in the direction of the width of the frame 10, and

[0124] The Y coordinate indicates the position of the pixel in the height direction of the frame 10,

[0125] v) a width L, hereinafter referred to as width L, and (e.g. measured in pixels)

[0126] The width value corresponding to the width of the bounding box B1-B3,

[0127] vi) a height A, hereinafter referred to as height A, and (e.g. measured in pixels)

[0128] the height value corresponding to the height of the bounding box B1-B3, and

[0129] vii) A shape factor value, referred to hereinafter as measured shape factor Rm for short, corresponding to the ratio between height A and width L (ie Rm=A / L).

[0130] Whenever information sets i)-vii) are determined for an object O1-O3, a check is then performed to determine whether frame 10 includes another object O1-O3 to be analyzed (decision step 1008). In the affirmative (output branch Y of step 1008), process 1000 is repeated starting from step 1007 to analyze another object O1-O3. Otherwise (output branch N of step 1008), all detected objects have been analyzed, and process 1000 is repeated starting from step 1001 to analyze a new frame 10 provided by camera 2.

[0131] In summary, in each iteration of the process 1000 , each object O1 - O3 present in the frame 10 acquired by the camera 2 is identified and the information given above in step 1007 is obtained.

[0132] The distance calculation process 2000 is adapted to calculate the distance according to a first quick estimate based on pre-stored parameters Kc for each type of recognizable object, and then to improve this estimate by taking into account two types of errors: "tilt error" and "rotation error", thereby obtaining a more reliable estimate.

[0133] The “tilt error” is due to the fact that the vertical axes of the camera 2 and the objects O1-O3 are considered to be misaligned, i.e. not parallel to each other (e.g. Figure 1(Schematically illustrated, with reference numerals N0-N3 used to indicate the vertical axis). For example, tilt error occurs when a skier assumes a tilted position relative to the camera (such as during a turn), so the corresponding bounding box is typically much wider and only slightly lower than the bounding box containing the skier in an upright position, and therefore appears straight relative to the camera. As a result, the diagonal of the bounding box of the tilted skier is substantially larger than the diagonal of the bounding box of the same skier standing at the same distance from the camera. This error leads to an underestimation of the distance. "Rotation error" is caused by the rotation of the object about its vertical axis, that is, the rotation of the object itself (about its vertical axis) relative to the front of the camera's field of view 2. For example, rotation error occurs when a wider and deeper object, for example, is rotated about its vertical axis so that it is no longer facing the camera directly. In this case, the bounding box surrounding the object is smaller than the bounding box of the same object when facing the camera. This leads to an overestimation of the actual distance. Conversely, when an object that is deeper than its height (such as a bus) is rotated about its vertical axis relative to the camera, the distance is underestimated.

[0134] For each object O1 - O3 identified in frame 10 , process 2000 receives information calculated by process 1000 that includes the type of object identified (step 2001 ).

[0135] Based on the information provided by process 1000 , a reference size of bounding box B1 - B3 , preferably a diagonal D of bounding box B1 - B3 , is determined (step 2002 ).

[0136] The distance of the object is estimated by comparing the diagonal D of the bounding box B1-B3 surrounding the identified object with at least one corresponding comparison dimension stored in the system. Preferably, the at least one comparison dimension is associated with a corresponding bounding box surrounding an object of the same type at a known distance. Even more preferably, the reference object used to define the comparison dimension is depicted with a vertical axis parallel to the vertical axis of camera 2 and with the front face of the reference object (e.g., the front side of a person's body, the front of a vehicle, etc.) arranged perpendicular to the plane of frame 10 including the reference object.

[0137] Specifically, the distance of the identified object is calculated as a function of the diagonal D of the bounding box B1-B3 and a constant value or coefficient associated with the type to which the identified object belongs (step 2003). Advantageously, for each recognizable object type, the portable device 3 stores a corresponding constant value determined as a function of the object type and one or more characteristic parameters of the camera (e.g., focusing aperture). In an embodiment, the distance Z is determined by means of the following relationship:

[0138]

[0139] Where D is the diagonal calculated at step 2002, and Kc is a constant value associated with object types O1-O3 identified from pre-stored constant values ​​for each object type O1-O3 and characteristic parameters of camera 2. In an embodiment, each constant value Kc is independently determined during a system calibration phase, starting from a known object in a vertical and frontal position to camera 2 at a known distance. In particular, applicants have determined that the coefficient Kc that relates the distance between objects O1-O3 and camera 2 to the value of the reference diagonal Dr of bounding box B1-B3 depends on the field of view associated with camera 2 and the measurement units (e.g., pixels) of the bounding box size B1-B3, i.e., per viewing angle. For example, in the case of a person, Kc is approximately 1170 ± 15%, considering that camera 2 has a vertical viewing angle of 62° and a horizontal viewing angle of 49°, and the reference bounding box has a height value A between approximately 0 and approximately 815 pixels and a width value L between approximately 0 and approximately 600 pixels, i.e., 13 pixels per viewing angle.

[0140] For example, to establish the value of the Kc parameter for an object type "person / human being", given a camera with a given viewing angle and given horizontal and vertical resolution, a person is photographed with camera 2 at a known distance (e.g. 2 meters) in a frontal and upright position relative to the camera. This image is sent to processing means 3, which recognize the presence of the person and generate a bounding box with a height of A units and a width of L units. The diagonal D of the bounding box is then calculated. For example, let's assume that diagonal D is equal to 10. If the same person is already 1 meter away, the value of diagonal D will be calculated, in this case 20, and Kc will be set equal to this value.

[0141] Advantageously, the generation of the Kc parameters for each object type is performed only once, and the parameters are supplied to one or more systems according to the present invention. In this case, the camera used to acquire the sample image may have different characteristic parameters (e.g., aperture angle, horizontal and vertical resolution, shape factor, etc.) than camera 2. Therefore, it is intended to compensate for any differences in the characteristic parameters to avoid introducing errors in the Kc parameter values ​​due to differences between the cameras.

[0142] Then, when the camera 2 captures a person at an unknown distance, the portable device 3 identifies the person-object type in the image and defines a corresponding bounding box and its dimensions in the same units as those used to define Kc. Based on these values, we will calculate the diagonal D from the bounding box, which in this case is equal to 5. Recalling the pre-stored Kc value for the object type "person", we apply formula (1) to obtain an estimated distance Z of 4 meters.

[0143] In order to increase the reliability of the estimate, it is preferred to calculate a default value for Kc for each object type based on an average of multiple detections using different objects of the same type and / or at a number of different known distances. For example, in the case of people, Kc can be established based on an average of Kc results obtained by framing different people with different body sizes, heights, arm positions, but representative of those whose distances will subsequently have to be estimated. It is also possible to determine several Kc constants for a given object type to be used for various diameter size ranges. Since, for collision avoidance, not only the absolute distance but also the change in distance over time is relevant, an error introduced by, for example, identifying an adult when the object being framed is actually a child (an event that leads to an underestimation of the distance) has no effect on the determination of whether it is approaching.

[0144] The final calculation of the estimate is then performed.Initially, pre-stored data relating to the identified object type is retrieved (step 2004).

[0145] In the non-limiting implementation form considered, the pre-stored data includes the following benchmarks:

[0146] - expected shape factor Ra,

[0147] - the expected depth parameter Pa, and

[0148] - The expected verticality or tilt parameter Va.

[0149] Specifically, in the shape implementation, the expected shape factor Ra is the ratio of the height to the width of a bounding box enclosing a reference object of the same type as the object O1-O3 under consideration facing camera 2. Thus, for each recognizable object type, the expected shape factor Ra has an empirically predetermined decimal value. For example, the bounding box of a person standing upright with respect to the camera and facing directly towards it, with their arms in a natural position along the body, is approximately 3.5. In other words, the parameter is determined (for each type) based on an image of the reference object acquired with its vertical axis arranged parallel to the vertical axis of camera 2 and its front face arranged perpendicular to one of the image planes (i.e., the front face is facing camera 2). In the example considered, the expression "front face" refers to the main part or surface of a stationary object (e.g., the surface of a billboard, the outline of a plant), or, for a moving object (e.g., a car, a person, a bicycle, etc.), the part or surface of the object facing its direction of travel.

[0150] The expected depth parameter Pa provides an indication of the effect of a rotation about the vertical axis on the ratio between the height and the width of the bounding box of the object type to which the considered object O1-O3 belongs, depending on the depth of the object O1-O3. In the embodiment considered, the expected depth parameter Pa is a Boolean parameter and is defined as "false" if the shape of the object type is such that the width of the bounding box decreases as the object rotates relative to the direction in front of the camera 2 (zero angle), and is defined as "true" when the width of the bounding box increases in the case of such a rotation (for example, in the case of a bus).

[0151] Preferably, the value of the expected depth parameter Pa for each object type is determined by evaluating the change in the size of the bounding box of at least one reference object of each type in a plurality of different rotations within a range of rotation angles between zero degrees (i.e., the front face of the object is perpendicular to the focal length direction of the camera) and a maximum angle Am exceeding which the object is considered to have a movement direction that is not incident with the camera optical axis. For example, in the case of a collision avoidance application, the maximum angle Am is set to ±25° (i.e., Am=±25°). Thus, if the bounding box width increases when the reference object has a rotation angle w within the range between zero degrees and the maximum angle Am (0°≤w≤±Am), the value of the expected depth parameter Pa for the object type is set to true, or if the bounding box width decreases when the reference object has a rotation within the same range, the value of the expected depth parameter Pa for the object type is set to false.

[0152] In the case of an object type with "square" sides (such as a bus), the value of the depth parameter Pa of the object can be determined mathematically as follows. Given the maximum angle Am and a proportional parameter Rp equal to the ratio between the width and the depth of the object type under consideration, we have:

[0153] Pa = false if (cos(Am)+sin(Am)) / Rp<1, and (2)

[0154] Pa=true, otherwise (3)

[0155] Thus, for example, for the object type "bus" defined in a non-exhaustive manner, in order to have an average depth of 13 m, an average width of 2.50 m and therefore a proportional parameter Rp equal to 0.19 (Rp=0.19), and considering a maximum angle of 25° (Am=25°), we have:

[0156] cos(25°)+sin(25°) / 0,19=5.19

[0157] That is, a value greater than 1, so that for this type of object the depth parameter Pa is true.

[0158] However, for other types of objects with blunt outlines (in particular, humans and other animals), the formula may result in "false positives". In this case, as a demonstrated implementation form, it is intended to empirically establish the value of the expected depth parameter Pa, at least for object types with blunt outlines. For example, the empirical value can be calculated by measuring the value by framing an object of the object type under consideration, in a vertical position relative to the vertical axis of the camera, but rotated to an angle equal to the maximum angle Am, and then measuring the effect on the width of the bounding box relative to the same object facing directly towards the camera (i.e., with an angle of zero).

[0159] The expected tilt parameter or expected verticality Va is a Boolean parameter that takes the value "true" if it is certain that the vertical axis of the camera 2 is substantially parallel to the vertical axis of the object type O1-O3, and takes the value "false" otherwise. Generally speaking, the expected verticality parameter Va can only take the value true if a horizon correction algorithm is implemented or only when the camera is in a fixed position and horizontal, and also only when the object should not be tilted (for example, a car); otherwise, the value Va is assumed to be false in all cases. For example, the expected verticality parameter Va is considered true for a car photographed by a camera 2 equipped with horizon correction, while the expected verticality parameter Va is considered "false" for a skier or a cyclist, regardless of the camera 2, because these types of objects may often be tilted relative to the horizon.

[0160] Parameters are defined for each of the object types that can be recognized by the system 1 and are stored by the system 1. Preferably, the data is preloaded into the memory of the portable device 3. For example, the portable device 3 receives and directly stores the parameters Ra, Pa, and Va of each type of recognizable object obtained by a system equivalent to the system 1 in which the processing device 3 operates. The parameters Ra, Pa, and Va of each type of recognizable object are then pre-stored in the system 1. Alternatively, the portable device 3 acquires or receives a plurality of reference images, each of which includes at least one recognizable object located at a known distance from the camera 2, with a main face facing the camera and a vertical axis parallel to the camera axis, so that the parameters Ra, Pa, and Va can be determined for each type of recognizable object.

[0161] For simplicity of illustration and not limitation, it is useful to define 10 categories based on the values ​​of a set of parameters. Each type of identifiable object may correspond to a corresponding category as follows:

[0162] - Category 1: Ra>1, Pa=false and Va=false,

[0163] Includes a front portion that is taller than it is wide and has a small depth (i.e., smaller than the other dimensions), can lean (such as a human skier) or cannot lean (such as a tree) but

[0164] Objects without horizon correction;

[0165] - Category 2: Ra>1, Pa=false and Va=true,

[0166] Includes objects with a frontal portion that is taller than it is wide and of small depth, which maintain their vertical axis substantially perpendicular to the ground (e.g. trees). This category is only available in cameras equipped with a horizon correction function.

[0167] are definable in positive systems (otherwise these objects fall into category 1);

[0168] - Category 3: Ra>1, Pa=True and Va=False,

[0169] Includes objects with a front portion or front that is taller than it is wide and a depth greater than it is high, which can be tilted about its vertical axis (e.g., motorcycles, bicycles) or

[0170] Cannot tilt (e.g. a truck with a trailer) but no horizon correction;

[0171] - Category 4: Ra>1, Pa=True and Va=True,

[0172] Includes objects with a frontal portion that is taller than it is wide or a frontal portion and a depth greater than it is high, said objects maintaining their vertical axis substantially perpendicular to the ground (e.g. trailers). This category is only definable in systems equipped with horizon correction (otherwise these objects fall into

[0173] Category 3);

[0174] - Category 5: Ra<1, Pa=false and Va=false,

[0175] Includes objects with a frontal portion that is wider than it is tall or a frontal portion and a small depth (few moving objects fit this specification), which may be tilted or may not

[0176] Tilt but no horizon correction;

[0177] - Category 6: Ra<1, Pa=false and Va=true,

[0178] Includes objects with a frontal portion that is wider than its height or a frontal portion and small depth (few moving objects meet this specification), which cannot be tilted and have

[0179] Horizon correction (otherwise these objects fall into category 5);

[0180] - Category 7: Ra<1, Pa=True and Va=False,

[0181] Includes objects with a frontal portion or front that is wider than its height and with a large depth (such as sports cars, speedboats; where depth is usually the longest dimension), which can be tilted (speedboats) or cannot be tilted (sports cars) but without horizon correction;

[0182] - Category 8: Ra<1, Pa=True and Va=True,

[0183] Includes objects with a frontal part or frontal face that is wider than its height and with a large depth, which cannot be tilted (like sports cars). This category can be defined only in systems equipped with horizon correction (otherwise these objects fall into category 7);

[0184] - Category 9: Ra≈1 (e.g. Ra=1±15%), Pa=false, independent of Va,

[0185] comprising objects having a substantially square front portion or front face and a small depth, the presence of

[0186] There are very limited circumstances in which objects of this type can move;

[0187] - Category 10: Ra≈1 (e.g. Ra=1±15%), Pa=true, independent of Va,

[0188] Included are objects having a substantially square front portion or front face and having a large depth (such as certain minivans).

[0189] It will be clear that the likelihood of assigning an object type to one of categories 2, 4, 6 or 8 also depends on whether a horizon correction algorithm as described above is used (or on the certainty that the camera is always horizontal), as this facilitates the pre-determination of the Va value for each object type.

[0190] Similarly, it will be clear to the skilled person that the tolerance of ±20% (more preferably ±15%) indicated above, defining the range of values ​​for Ra approximately equal to 1, is not to be understood in a limiting sense. On the contrary, in other embodiments, the tolerance value takes different values, for example the modulus of the tolerance value may take any value between 0 and 20% or between 0 and 15%, as required depending on the specific application.

[0191] In one embodiment, an association between each object type that can be detected by system 1 and the corresponding categories 1-10 mentioned above is predefined. In this case, the portable device 3 stores a data structure, such as a lookup table, a database, etc., in which member categories are defined for each type of object that can be recognized by the system.

[0192] Therefore, for each of the considered objects O1 - O3, its category (i.e., object type) to which it belongs is identified based on the pre-stored parameters (step 2005). Alternatively, only the category to which each recognizable object type belongs is pre-stored, and each object O1 - O3 is associated with the corresponding category based on its type.

[0193] In series or in parallel, the measured shape factor Rm and the expected shape factor Ra are compared to define whether the shape factor Rm is greater than, less than, or equal to the expected shape factor Ra (step 2006).

[0194] Based on the assigned category (at step 2005) and the comparison result between the measured and expected shape factors (from step 2006), it is defined whether the shape factor R is to be corrected and, if so, how to correct it (decision step 2007).

[0195] In the considered embodiment, as described below, what follows is a decision on whether to apply the correction and the selection of the specific correction to be applied. Optionally, the method further includes reporting when the detection of the object is found to be unreliable or ignored and the degree of reliability or indication of the final result provided by method 2000 even when the correction is applied.

[0196] In the considered embodiment, based on the category 1-10 assigned to the object and the comparison between the measured shape factor Rm and the expected shape factor Ra, the need to apply a correction and the selection of the correction to be applied to the detection of each recognized object O1 - O3 are determined.

[0197] Specifically, if Rm = Ra ± x%, where x is a predetermined tolerance value, such as x = 15, and more generally 0 ≤ x ≤ 20 (output branch N of step 2007), then the calculation of the diagonal Dr of the bounding boxes B1 - B3 of the objects O1 - O3 is not corrected (step 2008).

[0198] Otherwise, if the objects O1 - O3

[0199] a. belong to category 1 and Rm < Ra, or

[0200] b. belong to category 7 or 8 and Rm > Ra:

[0201] Apply the A correction described below (step 2009, output branch A from step 2007).

[0202] If objects O1 - O3

[0203] a. belong to category 1 and Rm > Ra,

[0204] b. belong to category 2 and Rm > Ra,

[0205] c. belong to category 4 and Rm < Ra,

[0206] d. belong to category 6 and Rm > Ra,

[0207] e. belong to category 7 or 8 and Rm < Ra,

[0208] f. belong to category 9 and Rm > Ra, or

[0209] g. belong to category 10 and Rm < Ra:

[0210] Apply the B correction described below (step 2010, output branch B from step 2007) in the calculation of the diagonal Dr of the bounding boxes B1 - B3 of objects O1 - O3.

[0211] If objects O1 - O3

[0212] a. belong to category 3 and Rm < Ra, or

[0213] b. belong to category 5 and Rm > Ra:

[0214] Apply the C correction described below (step 2011, output branch C from step 2007) in the calculation of the diagonal Dr of the bounding boxes B1 - B3 of objects O1 - O3.

[0215] If objects O1 - O3

[0216] a. belong to category 2 and Rm < Ra,

[0217] b. belong to category 3 and Rm > Ra,

[0218] c. belong to category 4 and Rm > Ra,

[0219] d. belong to category 5 and Rm < Ra,

[0220] e. belong to category 6 and Rm < Ra,

[0221] f. belong to category 9 and Rm < Ra, o

[0222] g. belong to category 10 and Rm > Ra:

[0223] No correction is performed and the signal D described below is generated (step 2012 , output branch D from step 2007 ), and method 2000 is repeated starting from step 2001 to determine the distance between camera 2 and another object O1 - O3 , information of which was received from process 1000 .

[0224] An "A correction" is a tilt (or inclination) error correction. Preferably, when an A correction is applied, a high reliability indication of the data thus corrected is generated.

[0225] A "B correction" is a rotation error correction. Preferably, when a B correction is applied, a high reliability indication of the data thus corrected is generated.

[0226] "C correction" is a combined correction obtained by averaging between correction A and correction B. Preferably, when the C correction is applied, an indication of low reliability of the correction data is generated.

[0227] "Signal D" is an indication of low reliability of the received data. For example, a D signal indicates that the input data related to a particular object detection and / or its bounding box are unreliable because they define conditions that should not occur in reality.

[0228] In a non-limiting manner, the expressions: "report this correction has a low / high degree of reliability", "report detection as unreliable", and "report shape factor as reliable" (and variations thereof) are intended to generate data (e.g., percentage values) that include information about the reliability of the results provided by process 2000.

[0229] refer to Figure 5 In the example of FIG, the correction of the tilt error (i.e., correction A) involves determining the diagonal Da of a bounding box Ba (i.e., a rectangle with width La and height Aa) having a shape factor equal to the expected shape factor Ra, the vertices of which lie on the perimeter of a bounding box Bm with a measured shape factor Rm and diameter Dm, as Figure 5 As shown qualitatively in the example of (where Ra = Aa / La and Rm = Am / Lm). Figure 5 Correction A involves determining the ratio between the diagonal Dm of the bounding box Bm and the diagonal Da of the bounding box Ba (i.e., a rectangle with width La and height Aa) based on the measured shape factor Rm and the expected shape factor Ra. This is a problem that can be solved by applying geometric / triangulation techniques (not described here for the sake of brevity). Applying correction A, the reference diagonal Dr is essentially equal to the diagonal Da of the bounding boxes surrounding objects O1-O3 at the same distance but straight from the camera.

[0230] For example, consider a case where execution of method 1000 provides data for a bounding box Bm containing an object O1 of type "human," and such bounding box Bm has a measured shape factor Rm of 2.5 (Rm=2.5), which is therefore different from the expected shape factor Ra of 3.5 for the type "human" (Ra=3.5). In such a case, knowing the measured shape factor Rm and the expected Ra, one skilled in the art can determine, by applying conventional geometry / triangulation techniques, that the only possible value for the ratio between diagonals Dm and Da is, in this case, approximately 1.064943. Thus, the diagonal Da of the bounding box Ba is calculated based on the diagonal Dm of the bounding box Bm and the calculated ratio between diagonals Dm and Da (i.e., 1.064943). Similarly, it will be clear to one skilled in the art that the tilt angle α of the considered object O1 "human" relative to the camera is approximately 7.352370°, which is the only tilt value that the vertical axes Nm and Na of the two bounding boxes Bm and Ba can have given Rm and Ra.

[0231] Regarding correction B (i.e., correction for rotation error), it compensates for the width of the bounding box of objects O1-O3 when modified by the rotation of objects O1-O3 around their vertical axis. In this case, the corrected width is calculated as the ratio between the height A of the bounding box B1-B3 and the expected shape factor Ra (i.e., =A / Ra), and therefore the relative diagonal. By applying correction B, the reference diagonal Dr substantially corresponds to the diagonal of the bounding box with height A and corrected width L'.

[0232] Correction C (i.e., compensation for rotation and tilt errors) requires that the corrected diagonal be calculated as the average of the first diagonal calculated by correction A and the second diagonal calculated by correction B. By applying correction C, the reference diagonal Dr is essentially equal to the average of the two diagonals calculated for objects O1-O3 at the same distance, straight in front of the camera and without rotation.

[0233] Downstream of steps 2008 - 2011 , the value of the reference diagonal Dr associated with the correct bounding box of the considered object O1 - O3 is available.

[0234] Next, fine distance values ​​(hereinafter referred to as distance Zr) between the objects O1 - O3 and the camera 2 (ie, the skier 1 ) are calculated based on the reference diagonal line (step 2013 ).

[0235] In the embodiment considered, the distance Zr is determined with the aid of the following relationship:

[0236]

[0237] where Dr is the reference diagonal calculated at steps 2008 - 2011 and Kc is a pre-stored constant value associated with the type of object O1 - O3 considered and the characteristic parameters of camera 2 .

[0238] After calculating the distances Zr of the objects O1-O3, a set of output data related to the objects (hereinafter referred to as IO outputs) is generated (step 2014). In the non-limiting example considered, the IO outputs include the following data:

[0239] i) a timestamp ts associated with the analyzed frame 10 comprising the objects O1-O3,

[0240] ii) the type of object being identified,

[0241] iii) X, Y coordinates of the corners of the corresponding bounding boxes B1-B3,

[0242] iv) width L, L' or L",

[0243] v) height A or A',

[0244] vi) distance Z, and

[0245] vii) Distance Zr.

[0246] Optionally, also provided:

[0247] viii) The confidence level, ie, the reliability indicator, of the estimated distance Zr determined at steps 2008-2011.

[0248] Finally, process 2000 is repeated through step 2001 to determine the distance between camera 2 and another object O1 - O3 about which information was received from process 1000 .

[0249] Alternatively, the IO output may include data relating to more than one object. For example, each IO output provided by process 2000 includes data associated with objects O1 - O3 detected in the same frame 10 acquired by camera 2 (and therefore associated with the same timestamp).

[0250] In the non-limiting embodiment under consideration, the IO output generated by process 2000 is used by process 3000, which determines whether one or more of the identified objects is approaching too close to camera 2 and, therefore, too close to the user. Preferably, process 3000 is adapted to compare data sets obtained from analysis of a sequence of frames 10 to detect one or more approaching objects and provide a corresponding warning signal to user 1. Optionally, process 3000 is configured to discard or limit the weight of distance estimates associated with low or substantially zero confidence levels considered in steps 2005-2011.

[0251] For example, process 3000 ( Figure 6 Flowchart 2000 (see flowchart thereof) receives the IO output generated by process 2000 (step 3001). Through analysis of the IO output, the change in distance between camera 2 and each detected object O1-O3, which has two or more consecutive frames IO in which the object O1-O3 appears, is evaluated (step 3002). In other words, the change in distance over time between camera 2 and each object O1-O3 is evaluated. This evaluation is used to detect one or more objects O1-O3 where the distance between object O1-O3 and camera 2 has decreased below a threshold, or alternatively, where the approach speed of object O1-O3 to camera 2 is greater than a threshold (decision step 3003). When no distance or speed change exceeding the corresponding threshold is detected (output branch N of step 3003), the process is repeated through step 3001 to analyze new IO output information. Otherwise, when the distance or velocity change of one or more objects O1-O3 exceeds a threshold value (output branch Y of step 3003), the spatial position of such one or more objects is determined based on the IO output (step 3004), and accordingly, an alarm signal is generated indicating the presence of the approaching objects O1-O3 and the approaching direction thereof (step 3005). For example, the alarm signal provides stereo sound using a tone having a pitch that is a function of the distance velocity change of the approaching objects O1-O3, a total volume defined by an inverse function (e.g., inversely proportional) of the distance between the camera 2 and the objects O1-O3, and right / left volume that is a direct function of the relative position of the approaching objects O1-O3 (the relative position being a function of the X and / or Y coordinates of the objects O1-O3).

[0252] The process is repeated starting from step 3001 , with each new receipt of an IO output being provided by process 2000 .

[0253] However, it is clear that the above examples should not be interpreted in a limiting sense and that the invention thus conceived is susceptible of numerous modifications and variations.

[0254] It will be apparent to those skilled in the art that the systems and processes according to the forms of the embodiments described above are not limited to the scope of skiing activities, which are merely examples of applications of the present invention. Rather, the systems and processes described above can also be used to increase the safety of other activities such as cycling, motorcycling, and other similar activities.

[0255] More generally, it will be apparent that the systems and processes described above enable easy and efficient determination of the distance of one or more objects detected by a single camera, regardless of the subsequent use of this information.

[0256] In an alternative embodiment of the object recognition process, a confidence value is calculated for each recognized object in a generic frame and checked to see if the confidence value is equal to or greater than a confidence threshold (e.g., 85%). In the following step, any objects with a confidence value below the confidence threshold are ignored. If all objects have been discarded, the entire frame is ignored.

[0257] Additionally or alternatively, the 1000A process for candidate object recognition includes a sub-process for checking for "out-of-bounds" of objects. In detail, for each object O1-O3 starting with the first object O1, it is verified whether the object O1-O3 under consideration included in the frame is only partially depicted in the frame 10 ( Figure 7 10), that is, whether there is an "out-of-edge" of the object O1-O3 outside the frame 10. In one embodiment, an out-of-edge of the object O1-O3 is verified if at least two of the coordinates of the angles X, Y, the coordinates of the maximum width X+L, and the coordinates of the maximum height Y+A are within a predetermined threshold distance from the corresponding frame reference edge 11-14 of the frame 10 (that is, from the boundary of the field of view (FOV) of the camera). For example, the threshold distance from the edge is set to a value between 2% and 6% of the size of the reference edge 11-14 considered in the frame 10, preferably equal to 5%. In detail, in each frame 10, where the width value x increases from left to right and the height value y increases from top to bottom, a protrusion appears if at least two of the following conditions are true: the angular coordinate X is at a distance from the right edge 13 of the frame 10 that is less than or equal to a threshold distance of the edge 13, the angular coordinate Y is at a distance from the bottom edge 14 of the frame 10 that is less than or equal to a threshold distance of the edge 14, the maximum width coordinate X+1 is at a distance from the left edge 11 of the frame 10 that is less than or equal to the threshold distance of the edge 11, and the maximum height width Y+A is at a distance from the top edge 12 of the frame 10 that is less than or equal to the threshold distance of the edge 12.

[0258] If overflow of the object O1-O3 under consideration is identified (output branch Y of step 1009), the object O1-O3 that is only partially depicted in frame 10 is disregarded and a check is performed to determine whether there are other objects O1-O3 depicted in frame 10 to be considered. In other words, the frame 10 associated with the object O1-O3 for which overflow was verified is disregarded. When overflow of the object O1-O3 outside of frame 10 is not verified (output branch N of step 1009), information regarding the object O1-O3 is defined. In other words, the operation proceeds to step 1007 described above. Furthermore, one or more steps of the same process 1000 / 1000A, 2000, or 3000 and / or two of the processes described above (e.g., 1000 / 1000A and 2000, or 3000 and 2000) may be performed in parallel with one another or in an order different from that presented above. For example, two or more instances of process 2000 may be performed in parallel, each for evaluating the distance of a different identified object. Similarly, one or more optional steps may be added to or removed from one or more of the processes depicted above.

[0259] For example, in an alternative form, in the case where an object "overflows" towards a single edge of the frame, data is not discarded, but starting from the expected shape factor of the identified object, the width or height of the bounding box of the affected object is estimated in a manner similar to that described above in the case of corrections due to rotation of the object about its vertical axis.

[0260] In other embodiments, at least the first estimation of the distance between the object and the camera is performed using techniques different from those described above. For example, in an embodiment with higher computational / energy consumption, a segmentation of the identified objects within each bounding box is performed and then the pixels representing the object are counted. The distance is estimated using a formula according to which the distance is inversely proportional to the number of pixels in the object. In an alternative, simpler embodiment, the distance between the object and the camera is estimated based on the size of the bounding box area; furthermore, the distance is inversely proportional to the size of the bounding box area.

[0261] Of course, all the details may be replaced by other technically equivalent elements.

[0262] For example, although the description relates to a system comprising a camera and a portable device, nothing prevents the system from being made as a single monolithic device or from providing different processing devices (such as dedicated processing devices).

[0263] In summary, the materials used, as well as the possible shapes and sizes of the above-mentioned devices, equipment and terminals can be any according to the specific implementation requirements without departing from the scope of protection of the accompanying claims.

Claims

1. A system (2, 3) for measuring the distance of an object (O1-O3) from the system, the system comprising a single camera (2) and a processing device (3) connected to the camera (2), wherein the camera (2) acquires an image, and The processing device (3) is: For each object type in a plurality of identifiable object types, at least one parameter among the following is pre-stored: the expected shape factor between the height and width of the bounding box of at least one reference object, expected depth data determined as a function of a change in a measured form factor of the reference object as the rotation angle of the front face of the reference object relative to the camera (2) changes, and expected verticality data determined based on at least one reference object of the object type, the position of the camera (2) and / or the presence / absence of horizon correction of the acquired images from the camera (2), receiving the acquired image, executing an image recognition algorithm that, for each identified object in the acquired image: identifying the type to which the identified object belongs among a plurality of identifiable object types, and defining a bounding box surrounding the identified object, determining the reference dimensions of the bounding box, Correcting the bounding box of the identified object by applying a correction criterion said reference size, wherein said correction criterion is selected as a function of a value of at least one pre-stored parameter associated with said type of said identified object and at least one characteristic of said bounding box of said identified object, and As the reference size thus corrected and at least one stored in the system The distance to the identified object is calculated by a function of at least one comparative size, wherein at least one comparative size is a function of a known object of the same type located at a known distance. The corresponding bounding box of the object is associated.

2. The system (2, 3) according to claim 1, wherein Determining the at least one pre-stored parameter per object type: Identify reference objects in reference images, and A corresponding reference bounding box is created, wherein the vertical axis of the reference object is parallel to the vertical axis of the camera (2) and the front face of the reference object is positioned vertically relative to the plane of the image.

3. The system (2, 3) according to claim 1 or 2, wherein: The processing means (3) determines that the expected depth data of each object type having a square outline is logical data having a true value if: cos(Am)+sin(Am) / Rp<1 wherein Am is the maximum trajectory angle of the object type relative to the camera (2), and Rp is the average ratio between the width and depth of the object type, and otherwise the expected depth data is determined to have a false value.

4. System (2, 3) according to any one of the preceding claims, wherein The processing means (3) corrects the reference size by applying a correction criterion for tilt errors, calculating a corrected bounding box having the expected shape factor and corners located on the sides of the bounding box of the identified object.

5. System (2, 3) according to any one of the preceding claims, wherein The processing device (3) corrects the reference dimension by applying a correction criterion for the rotational error and calculating the corrected width of the bounding box as the ratio between the height of the bounding box enclosing the identified object and the expected shape factor of the type of the identified object.

6. System (2, 3) according to claims 4 and 5, wherein The processing device (3) corrects the reference dimension by applying a correction criterion for the composite error and calculating the corrected reference dimension of the bounding box as the average of a first reference dimension calculated by applying the correction of the tilt error to the bounding box enclosing the identified object and a second reference dimension calculated by applying the correction of the rotational error to the bounding box enclosing the identified object.

7. System (2, 3) according to any one of the preceding claims, wherein The processing device (3) pre-stores parameters of an expected shape factor, expected depth data, and expected perpendicularity data for each object type and selects which correction to apply to the reference dimension as a function of the values of the pre-stored parameters associated with the type of the identified object and the shape factor of the bounding box of the identified object.

8. System (2, 3) according to claims 4 and 7, wherein The processing device (3) identifies the need for correcting the tilt error when: Ra > 1, Pa = false, Va = false and Rm < Ra, or Ra < 1, Pa = true and Rm > Ra where Ra is the expected shape factor of the identified object of the type, Pa is the expected depth data, Va is the expected perpendicularity data, and Rm is the measured shape factor of the identified object.

9. System (2, 3) according to claims 5 and 7, wherein The processing device identifies the need for correction of the tilt error when: Ra > 1, Pa = false, Va = false and Rm > Ra, Ra > 1, Pa = false, Va = true and Rm > Ra, Ra > 1, Pa = true, Va = true and Rm < Ra, Ra < 1, Pa = false, Va = true and Rm > Ra, Ra < 1, Pa = true, and Rm < Ra, Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, with 0 ≤ x ≤ 20, Pa = false, and Rm > Ra, or Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, with 0 ≤ x ≤ 20, Pa = true, and Rm < Ra, where Ra is the expected shape factor of the identified object of the type, Pa is the expected depth data, Va is the expected perpendicularity data, and Rm is the measured shape factor of the identified object.

10. System (2, 3) according to claims 6 and 7, wherein The processing device identifies the need for composite correction when: Ra > 1, Pa = true and Va = false and Rm < Ra, or Ra < 1, Pa = false and Va = false and Rm > Ra, where Ra is the expected shape factor of the identified object of the type, Pa is the expected depth data, Va is the expected perpendicularity data.

11. System (2, 3) according to any one of claims 7 to 10, wherein [[ID= ​ ​ Ra < 1, Pa = false and Rm < Ra, Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, with 0 ≤ x ≤ 20, Pa = false and Rm < Ra, or Ra ≈ 1, preferably 1 - x% ≤ Ra ≤ 1 + x%, with 0 ≤ x ≤ 20, Pa = true and Rm > Ra.

12. The system (2, 3) according to any one of the preceding claims, wherein The processing device (3) at least pre - stores the expected shape - factor parameters, and at least one characteristic of the bounding box of the identified object includes the shape factor of the bounding box, and where the processing device (3) does not apply any correction to the reference dimension when: Rm = Ra ± x%, with 0 ≤ x ≤ 20, where Rm is the shape factor of the bounding box of the identified object calculated from the processing device, and Ra is the expected shape factor of the identified object of that type.

13. The system (2, 3) according to any one of the preceding claims, wherein The reference dimension is the diagonal of the bounding box of the identified object, and the comparison dimension is the diagonal of the bounding box surrounding a known object of the same type at a known distance from the system (2, 3), the known object having a vertical axis parallel to the vertical axis of the camera (2) and a front face parallel to the plane of the image.

14. The system (2, 3) according to claim 13, wherein The distance between the identified object and the system is calculated by means of the following formula: where D is the diagonal of the bounding box of the identified object, Kc is a constant value stored by the processing device (3) for each object type among the plurality of types, and Kc is a function of the characterization parameters of the object type and the camera (2).

15. System (2, 3) according to any one of the preceding claims, wherein The processing device (3) is adapted to calculate the reliability of each identified object and / or the reliability of the corresponding calculated distance, where each reliability is calculated as a function of at least one parameter associated with the type of the corresponding identified object, and preferably, is calculated as a function of the difference between the expected shape factor between the height and width of the bounding box of an object of the identified object type and the measured shape factor between the height and width of the bounding box of the identified object.

16. A method for measuring the distance between an object (O1 - O3) and a system (2, 3), the system comprising a single camera (2) and a processing device (3) connected to the camera (2), The method comprises the following steps: Acquiring an image by means of the camera (2), and By means of the processing device (3): For each object type among a plurality of recognizable types, pre - store at least one of the following parameters: The expected shape factor between the height and width of the bounding box of at least one reference object, The expected depth data determined as a function of the change in the measured shape factor of the reference object when the rotation angle of the front face of the reference object with respect to the camera (2) changes, and The expected verticality data determined based on at least one reference object of the object type, the position of the camera (2) and / or the presence / absence of horizon correction of the acquired image from the camera (2). receiving the acquired image, executing an image recognition algorithm that, for each identified object in the acquired image: identifying the type to which the identified object belongs among a plurality of identifiable object types, and defining a bounding box surrounding the identified object, determining the reference dimensions of the bounding box, correcting the reference size of the bounding box of the identified object by applying a correction criterion, wherein the correction criterion is selected as a function of a value of at least one pre-stored parameter associated with the type of the identified object and at least one characteristic of the bounding box of the identified object, and The distance to the identified object is calculated as a function of the reference size thus corrected and at least one comparative size stored in the system, wherein the at least one comparative size is associated with a corresponding bounding box enclosing a known object of the same type located at a known distance.

Images