Computerised system for measuring objects using artificial intelligence and augmented reality

Abstract

The present invention relates to a system for recognising and measuring the volume of wood in log stacks using artificial intelligence and augmented reality to identify and georeference the detected stack using mobile devices, such as smartphones or tablets, to capture the frames and provide the calculation of the volume of wood automatically and in real time, eliminating the need to store photos or videos.

Description

DESCRIPTIVE REPORT COMPUTERIZED SYSTEM FOR MEASURING OBJECTS USING ARTIFICIAL INTELLIGENCE AND AUGMENTED REALITY Field of innovation [ooi] This innovation belongs to the field of object recognition and measurement systems without physical scale conversion, more specifically, a computerized system for recognizing and editing the volume of wooden logs using artificial intelligence through deep neural networks and augmented reality as identification, location and processing tools. History of innovation

[0002] The volumetric calculation of wood, also called “cubing”, whether of standing trees or logs in storage parks, involves measuring the dimensions of the logs (diameter and length) or by volumetric estimation based on approximate methods, images or averages stipulated according to each species based on the age of the trees.

[0003] The commercialization of roundwood is traditionally done using volume measurements derived from the “external” surface of the wood pile, that is, the volume that includes wood and residual matter (air and other residues). Tradition has consolidated the use of the “stereo” unit to refer to the volume, in cubic meters, of a roundwood pile, based on measurements made on the “surface” of the log pile and on statistical estimates of the stacking factor, which is the ratio of the volume of the pile to the volume of wood. Recent automatic measurement systems based on laser mapping of the surface of timber piles have further reinforced the importance of “surface volume” information in the timber trade.

[0004] However, such methods are either very costly, in the case of physical measurement of the pieces, as they require intensive use of labor, or they are very imprecise, resulting in deviations that are sometimes very significant in determining the financial value linked to the volume of wood to be considered. In other words, current methods lack precision or manual activity that is costly in terms of time and human and financial resources.

[0005] Alternative cubing methods, such as measuring the volume displaced by logs submerged in water, a method known as xylometry, in addition to being impractical, are unfeasible if applied to trees that have not yet been cut.

[0006] Alternatives are then sought that can automate the calculation without compromising accuracy or generating more sources of uncertainty or costs that compromise its application.

[0007] Even existing digital solutions fall into the limitations of “photogrammetry”, which requires the use of a comparative physical reference to extract measurements.

[0008] In this vein, current tools such as artificial intelligence (AI) and augmented reality (AR) have proven useful in this automation process, making it possible to make measurements quick and practical through real-time frame analysis, eliminating the need to store photos or videos, as well as eliminating the need to use scales for referencing or converting real-world measurements. Discussion of the state of the art

[0009] Document KR20200125131 discloses a method and a system for measuring the thickness or volume of an image or an area in a 2D or 3D image using an artificial neural network and visualizing the measured thickness or volume, wherein the artificial intelligence (AI)-based image thickness measurement system comprises a model generation unit for generating information required to generate a model, a model learning and evaluation unit for learning data and evaluating the data; and a scanning and visualization unit for performing scanning and visualization based on the learned result, wherein the scanning and visualization unit three-dimensionally visualizes a result value of a thickness ratio of an image area of ​​the model learning and evaluation unit, measures the result value of the thickness ratio and a result value of a voxel distance (3) in an area, and combines and quantifies the measured values. [ooio] The solution taught in W02017125103 shows a method for displaying georeferencing information from a georeferencing database by means of a geoinformation system, wherein the georeferencing information comprises coordinates and sonar data corresponding to the coordinates and are assigned to individual voxels (3), such that a voxel (3) is a representation of one of the coordinates and the corresponding sonar data, and the voxel (3) or information derived from the voxels (3) can be displayed on a two-dimensional display, the sonar data being generated using a curved measurement curve, the method comprising the steps: - forming a measurement curve orthogonal to the curved measurement curve, - determining the voxels (3) corresponding to the orthogonals of the measurement curve, - combining the sensor data visualized in the voxels (3) such that side view information is provided, and displaying the side view information on a display of the geoinformation system. [ooii] CN112580641 provides teachings relating to an image feature extraction method and device, a storage medium and electronic equipment, and belongs to the field of artificial intelligence, wherein the method comprises the following steps: cutting an original image into a plurality of hypervoxel scales; calculating a superpixel feature for the superpixel of each scale to obtain the superpixel features of a plurality of scales; and merging the superpixel features based on the multiple scales to obtain the multi-scale features of the original image, where the technical problem of low degree of distinction of image features in related technology is solved.

[0012] Patent CN111429460 discloses an image segmentation method, an image segmentation model training device and a storage medium, wherein the image segmentation method comprises the steps of: acquiring an image to be segmented; extracting voxel features (3) from the image to be segmented; determining a first similarity between the voxel features (3) and the voxel reference values ​​(3) corresponding to different classes; the voxel reference value (3) is a model parameter obtained by performing model training using an image sample to be segmented; and when the first similarity between the target voxel feature (3) in the voxel features (3) and the voxel reference value (3) corresponding to the category meets the segmentation condition, segmenting an image block corresponding to the target voxel feature (3) in the image to be segmented.

[0013] In US2022058792 one can see the inventory management of wood and timber products with the continuous measurement and counting of individual items, which involves taking a picture of products using a smartphone or a tablet's built-in camera, processing the image data to detect individual items using artificial intelligence object detection methods, and using special algorithms to measure and calculate unit volume to present the user with a detailed description, measurement, count and a summary.

[0014] Patent US2020327653 provides a system based on a portable device configured to detect, count and measure logs in a pile by capturing images of the pile, creating a working image by joining parts of the image and identifying a contour indicating the outline of the pile in the working image and fitting ellipses to the faces of the logs in the working image, gathering information such as the number of logs, the volume of wood in the pile and the average diameter of the logs that can be made available for presentation to the user.

[0015] CN113256738 discloses a binocular parcel volume measurement method comprising using a binocular camera to acquire an image of a parcel to be detected, calculating a horizontal stripe disparity map at the center of the image, calculating a parallax map of a central vertical stripe of the image, detecting a parallax jump area to obtain 4 sets of parallax edge points, fitting 4 linear equations based on the sets of parallax edge points, rotating and adjusting two opposite sides about the center of each line segment to allow the opposite sides to be parallel, obtaining the height of the object to be measured, obtaining the length and width of the parcel to be detected, and calculating the volume of the parcel to be detected according to the length, width, and height of the parcel to be detected.

[0016] It is noted that, in the state of the art, there is no solution comparable to the present innovation, where the use of augmented reality associated with artificial intelligence and convolutional neural network in a recognition and measurement system, allows the volume of stacked logs to be assessed, even without the use of photos or videos to identify the logs. Description of figures

[0017] Figure 1 represents a pile of logs (3), the object of measurement by the innovation proposed here, with a 3D mesh formation (2) on its main transverse face. All data were captured by the spatial and geolocation sensors of the mobile device (1).

[0018] Figure 2 represents the discretization of the 3D mesh (2) performed on the face of the stack (3) by a mobile device (1) for a simplified geometry, primarily captured and the pre-identification of its coordinate centers (4).

[0019] Figure 3 shows the preparation of a sampling (5) of the total quantity of logs in the pile (3).

[0020] Figure 4 shows us the calculation of a normal vector (6) for each of the logs in the sampling group (5).

[0021] Figure 5 represents the calculation of the average of all normals (6) found in the sampling (5), transformed into a single vector (7) resulting from the previously calculated average, thus directing a main plane (8).

[0022] Figure 6 shows a mobile device (1) that, through a scanning movement, scans the main face of the log pile (3) to extract a random frame (9) and then send it for analysis by a deep neural network.

[0023] Figure 7 shows us the result of the analysis of the deep neural network that returns a bounding box (10) composed of two properly identified spatial coordinates for each log belonging to the pile (3) measurement object.

[0024] Figure 8 explains the analysis that the mobile device (1) performs when executing a raycasting technique (11) to identify 4 tangency points between the bounding box (10) and the logs that are the object of the proposed measurement.

[0025] Figure 9 represents the capture of data provided by the bounding box (10) and its center coordinate (4) of the log to perform calculations of the average horizontal and vertical distances.

[0026] Figure 10 shows us a mobile device (1) using its spatial and geolocation sensors to generate a simulated environment for the instantiation of 3D objects (12).

[0027] Figure 11 represents the projection and anchoring of the 3D objects (12) on the real and main face of the stack (3).

[0028] Figure 12 represents the capture of spatial and geolocation data by the mobile device (1) and the sending of several calculated diameter values ​​(13) for analysis in a deep neural network, thus returning the final diameter value (14).

[0029] Figure 13 shows us the update flow on screen (15) based on the best diameters (13) found by the network. deep neural beyond reporting flow and cloud synchronization. Description of innovation

[0030] This innovation operates based on a system that adopts a real-time frame identification and processing approach, resulting in the creation of a three-dimensional and spatially localized mesh. This approach enables the system to recognize the logs of wood through a deep neural network, which are then measured. The system also allows the quantification of the volume of the identified logs using dedicated software. This program captures the frames through the camera of a mobile device.

[0031] As illustrated in Figure 1, the system creates a detailed three-dimensional representation of the environment in which log measurements (3) will be performed by the mobile device (1). This mesh (2) is essential for mapping the space and allowing the logs (3) to be accurately virtualized. Augmented reality (AR) technology is employed to combine real-world data with virtual elements, making it possible to overlay measurement information onto real logs.

[0032] To generate the mesh (2), the system uses advanced sensors, such as geolocation sensors, which determine the exact position of the device in space, and spatial sensors (LiDAR), which emit laser pulses to measure distances and create a three-dimensional map of the environment. These sensors play a role fundamental in capturing accurate data, allowing the system to understand the full complexity of the environment's structure.

[0033] Figure 2 shows the 3D mesh discretization process (2). In this step, the system simplifies the measurement process by converting the face of the log pile (3) into a two-dimensional plane. This is important for detailed analysis of the logs, as it provides a reference surface on which measurements can be performed accurately. In this phase, the system identifies and isolates the surface of the log pile (3), distinguishing it from the surrounding environment. This step is crucial to ensure that measurements are specific to the logs, eliminating interference from other elements in the scene. The system also pre-identifies the coordinate centers (4) of each log present in the scene.

[0034] After the spatial location of the log pile (3) and its isolation from the environment in which it is located, and in order to simplify the measurement process as well as ensure representativeness, a random sampling (5) of logs was taken from the complete pile (3) as shown in Figure 3. This sample will serve as a basis for subsequent measurements and vector analyses.

[0035] After identifying each coordinate center (4) of the selected logs from the sampling group (5) represented by Figure 3, the system calculates the normals (6) of each of the sample logs as represented in Figure 4. The normals (6) represent directions perpendicular to the surface of each log, which is essential for accurately determining its dimensions.

[0036] As illustrated in Figure 5, it is possible to conceive of calculating an average of the normals found in the sampling (5) with the aim of obtaining a normal that significantly represents the median surface of the logs in the pile (3). In this sense, the normals of each log (6) identified in the sampling (5) are combined using a weighted average, resulting in the formation of a main work plane (8) from the calculated resulting vector (7), which becomes the central reference plane for the entire pile of logs (3). This work plane establishes a solid basis for the future measurement and anchoring of the diameters.

[0037] After the log identification phase, the detailed measurement process illustrated in Figure 6 begins. In this step, random frames (9) from the pile (3) are extracted and analyzed by a deep neural network, which identifies the logs and returns information about their extreme points.

[0038] As shown in Figure 7, the deep neural network also returns a bounding box (10) from the previously identified extreme points. From this same bounding box (10), relative points are identified and transformed into pixel coordinates in the image, allowing for more accurate identification of each log found in the pile (3).

[0039] Using a light ray simulation technique to measure distance and collision of objects in predefined vector directions called raycasting (11), the application launches a vector from the mobile device (1) thus identifying tangency points between the bounding box (10) and the log to be measured as shown in Figure 8.

[0040] Horizontal and vertical dimensions (Lx and LY) of the bounding box (10) are calculated using per-pixel measurements and the information from the previously defined work plane (8). These points allow us to calculate the diameter values ​​of all the logs mapped in the pile (3). The system at this stage also recalculates the center coordinates (4) of each sample and sends the data back for analysis via deep neural network.

[0041] In Figure 10, we can illustrate the logs that are then virtualized in a simulated 3D environment, where each log is virtually represented based on the measurements taken.

[0042] In Figure 11 we can understand the process of inserting 3D objects (12) into the real environment via augmented reality technology, allowing a perfect overlap between the virtual and real logs on the 3D mesh (2), belonging to the main face of the pile (3).

[0043] Once the software has already stored the data of the center coordinates (4) of each log in the pile (3) as well as the pre-values ​​of their respective diameters, the application uses the spatial positioning and geolocation sensors to capture new values ​​of each diameter within its bounding box (10) in a continuous process. This process can be visualized in Figure 12. The system performs analyses to determine whether it is necessary to update the measurements or exclude any logs from the process. A neural network evaluates the measurements and determines whether the diameter values ​​can be improved based on the device position and the user's perspective, thus returning the final diameter (14).

[0044] In Figure 13, we can see that the data collected (13) during the process is transferred to the user interface (UI) for analysis. The UI generates detailed reports that include information about log dimensions, statistics and other properties plotted on the interface (15) of the mobile device (1) in real time. Finally, the data is transferred together with global positioning information captured by the device's GPS sensor (1) and stored in a database for future reference. An automatic cloud synchronization functionality ensures that the data is available remotely and securely.

Claims

CLAIMS 1. COMPUTERIZED SYSTEM FOR MEASURING STACKED WOOD USING ARTIFICIAL INTELLIGENCE AND AUGMENTED REALITY characterized by comprising the formation of a 3D mesh (2) to identify the location of the faces of the stacked logs (3) by the dispersion of the center coordinate points (4) and their consequent spatial position, with the artificial intelligence identifying the logs (3) in real time, analyzing the distance between the mobile device (1) and the logs (3), and relating the projected area (12) of the disk (10) of each face with the length of the logs (3) to calculate the volume of wood comprised in said stack (3).

2. COMPUTERIZED SYSTEM FOR MEASURING OBJECTS USING DEEP NEURAL NETWORKS AND AUGMENTED REALITY, such as that of claim 1, characterized by, alternatively, using photogrammetry to generate a mesh of points and the consequent cloud of points by superimposing frames generated through perspective and depth of the captured frames.

3. COMPUTERIZED SYSTEM FOR MEASURING OBJECTS USING ARTIFICIAL INTELLIGENCE AND AUGMENTED REALITY, such as that of claim 1, characterized by being capable of identifying the species of wood evaluated by the deep neural network in said system through standardized peculiarities and their intersection with the UV map.

Images