System and method for correcting paving mat defects

Abstract

A method includes receiving sensor data indicative of a paved surface, and identifying a defect associated with the paved surface based at least in part on the sensor data. The method also includes determining that the defect belongs to a defect type based on determining that a value associated with the defect is within a range of values ​​associated with the defect type. The method further includes generating instructions associated with the defect that, when executed by a machine, at least partially remedy the defect. The method also includes providing the instructions to an electronic device via a network.

Description

Technical Field

[0001] The present invention relates to a control system for a paving machine. More particularly, the present invention relates to a control system configured to identify paving mat defects and generate instructions to correct the identified paving mat defects. Background Art

[0002] A paving machine, such as an asphalt paver, is a self-propelled construction machine designed to receive, convey, distribute, shape, and partially compact paving material. Such a paver receives heated paving material (e.g., asphalt) into a receiving hopper at the front of the paver. The heated asphalt material in the hopper is transported to the rear of the paver by a conveyor positioned at the base of the hopper. The asphalt material is then distributed across the width of the paver by two opposing screws or augers. Finally, a scraper assembly located at the rear of the paver shapes and compacts the asphalt material into an asphalt surface, called a "pad."

[0003] Paving machines have many components that are capable of interacting with each other and with the paver's surroundings, and are configured in a variety of ways depending on the type of paving material being applied and the characteristics of the environment. To name a few examples, if a portion of the paver fails, there is an error in the paver's computing system, or the paver's settings are incorrect or inappropriate for the conditions at hand, the resulting mat may have one or more defects. Defects in the mat (paving surface) may require repair of the mat, or in some cases, complete removal and replacement of the mat, which consumes valuable time and resources. While much of the operation in the paving process relies heavily on the training, skill, and experience of the paver operator, the paver operator or other personnel often do not have enough experience to identify every possible defect that may occur in a mat, and / or how to address and mitigate the defect once it has been identified.

[0004] An exemplary system for determining road surface defects is described in U.S. Patent Application No. 2012 / 0218441 (hereinafter referred to as the '441 application). Specifically, the system described in the '441 application is configured to analyze infrared thermal images of road surfaces and diagnose defects present on the road. The '441 application further describes various preventive maintenance solutions. However, the system described in the '441 application requires an analyst to judge all images describing defects one by one to provide treatment conclusions and / or proposed treatment solutions. The accuracy of these conclusions depends on the experience and training of individual analysts, and these conclusions are prone to human error. Additionally, the conclusions provided by the analyst described in the '441 application depend on whether the analyst is available when the paving defect is identified, which may cause delays based on the analyst's bandwidth and / or scheduling in paving scenarios. Such delays reduce the overall efficiency of the system and / or paving operations.

[0005] Exemplary embodiments of the present invention are intended to overcome the above-mentioned drawbacks. Summary of the invention

[0006] In one aspect of the invention, an exemplary method includes receiving, with a controller, image data indicating a paved surface; and identifying, with the controller and based at least in part on the image data, a defect associated with the paved surface, the defect having a characteristic characterized by a value. This exemplary method also includes determining, with the controller, that the value is within a range of values ​​corresponding to a defect type; and determining, based at least in part on the value being within the range of values, that the defect belongs to the defect type. This exemplary method further includes generating instructions associated with the defect based at least in part on determining that the defect belongs to the defect type. In such a method, the instructions identify a proposed machine activity that, when performed by the machine on the paved surface, at least partially remedies the defect. This exemplary method also includes providing instructions to an electronic device via a network.

[0007] In another aspect of the invention, a system includes a sensing device and a system controller in communication with a controller of the sensing device. The system controller is configured to receive sensor data indicating a paved surface from the sensing device and, based at least in part on the sensor data, identify a defect associated with the paved surface, the defect having a characteristic characterized by a value. The controller is also configured to determine that the value is within a range of values ​​corresponding to a defect type and, based at least in part on the value being within the range of values, determine that the defect belongs to the defect type. The controller is further configured to generate instructions associated with the defect based at least in part on determining that the defect belongs to the defect type, wherein the instructions identify a proposed machine activity that, when performed by the machine on the paved surface, at least partially remedies the defect. The controller is further configured to provide the instructions to an electronic device via a network.

[0008] In yet another aspect of the invention, an exemplary method includes: receiving, using a controller and via a network, sensor data indicating a paved surface at a worksite from a sensing device located at the worksite; and identifying, based at least in part on the sensor data, a defect associated with the paved surface, wherein the defect has a characteristic characterized by a value. This exemplary method also includes determining, using the controller, that the value is within a range of values ​​corresponding to a defect type. This exemplary method further includes determining, using the controller, and based at least in part on the value being within the range of values, that the defect belongs to the defect type. This exemplary method further includes: generating, using the controller, instructions associated with the defect based at least in part on determining that the defect belongs to the defect type, wherein the instructions identify a proposed machine activity that, when performed by a paving machine on the paved surface, at least partially remedies the defect. This exemplary method further includes providing, by the controller, instructions to a paving machine via the network, wherein the instructions cause the paving machine to change a setting of a paving machine component associated with the proposed machine activity. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a perspective view of a paving machine including a machine controller in communication with a pad defect identification component according to an example of the present invention.

[0010] Figure 2 is a block diagram schematically illustrating a control system associated with a paving machine according to an example of the present invention.

[0011] Figure 3 is a schematic diagram of a paving machine depositing paving material on a paved surface according to an example of the present invention, wherein the mat is free of defects.

[0012] Figure 4 is a schematic diagram of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect according to an example of the present invention.

[0013] Figure 5 is a schematic diagram of a paving machine and associated components of the paving machine depositing paving material on a paved surface according to an example of the present invention, and dimensions that may be varied to adjust how the paving material is deposited on the paved surface.

[0014] Figure 6 is a schematic diagram of a sensor included on a paving machine that determines a distance to paving material according to an example of the present invention.

[0015] Figure 7 is another schematic illustration of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect in accordance with an example of the present invention.

[0016] Figure 8is yet another illustration of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect according to an example of the present invention.

[0017] Fig. 9 is yet another illustration of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect according to an example of the present invention.

[0018] Fig.10 2 is an additional schematic diagram of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect in accordance with an example of the present invention.

[0019] Fig.11 is a further illustration of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect according to an example of the present invention.

[0020] Fig.12 is another schematic illustration of an image of a mat having a defect on a paved surface and instructions that may be provided by a control system based on the defect in accordance with an example of the present invention.

[0021] Fig.13 is a flow chart depicting a method of generating instructions for remediating an identified paving mat defect according to an example of the present invention. DETAILED DESCRIPTION

[0022] Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. Figure 1 A paving machine 100 is shown for use, for example, in depositing asphalt, concrete, or other material on a work surface associated with a work site. Figure 1 As shown, paving machine 100 communicates with one or more computing devices 204 via network 206. The one or more computing devices 204 include a pad defect identification component, namely a pad defect ID component 214, which is configured to identify a defect in a paving mat and generate instructions that, when executed by paving machine 100, at least partially remedy the defect in the field using a crew of paving machine 100, a haul truck delivering paving material 110 to paving machine 100, etc. Although the following detailed description describes examples related to paving machine 100, it should be understood that the description is equally applicable to the use of the present invention in other machines.

[0023] The paving machine 100 includes a tractor section 102 supported on a set of ground engaging elements 104. The tractor section 102 includes a tractor frame 106, and a power source for driving the ground engaging elements 104. Although the ground engaging elements 104 are shown as continuous tracks, it should be understood that the ground engaging elements 104 can also be any other type of ground engaging elements, such as wheels, etc. In some cases, the power source is a conventional internal combustion engine running on fossil or mixed fuels, or in the example, an electric drive powered by an alternative energy source. The paving machine 100 includes a hopper 108 for storing paving material 110. The paving machine 100 also includes a conveyor system 112 for conveying the paving material 110 from the hopper 108 to other downstream components of the paving machine 100. For example, the paving machine 100 includes an auger assembly 114, which receives the paving material 110 supplied via the conveyor system 112 and distributes the paving material 110 on the paving surface 122. In some examples, the auger assembly 114 includes at least one main auger. In some cases, the auger assembly 114 includes a main auger and an auger extension coupled to the main auger via an auger bearing or other coupling component. Additionally, in some examples, the auger assembly 114 includes a main auger and an additional auger disposed opposite the main auger. In such examples, the main auger and the additional auger are configured to distribute paving material 110 over substantially the entire width of the paving machine 100. The paving machine 100 also includes a traction arm 116 that couples a height-adjustable scraper section 118 to the tractor section 102 to spread and compact the paving material 110 into a pad 120 on a paving surface 122. The traction arm 116 is actuated by a hydraulic actuator, an electric actuator (not shown), and / or any other type of actuator as required by the application. In an example, any of the ground engaging elements 104 , hopper 108 , conveyor system 112 , auger assembly 114 , traction arm 116 , and / or scraper section 118 may receive instructions from the pad defect ID component 214 to adjust settings such as speed, height, etc., to remedy a defect detected in the pad 120 .

[0024] Further references Figure 1 , an operator station 124 is coupled to tractor section 102. Operator station 124 includes a control console 126 and other levers or controls (not shown) for operating paving machine 100. For example, control console 126 includes a control interface 128 for controlling various functions of paving machine 100. Control interface 128 may include an analog, digital, and / or touch screen display, and such control interface 128 is configured to display, for example, instructions for remedying defects of mat 120 according to the present invention when executed by paving machine 100. Control interface 128 also supports other related functions, including, for example, sharing various operating data with one or more other machines (not shown) operating in coordination with paving machine 100.

[0025] As shown, the operator station 124 includes a top panel 130. A communication device 132 and / or a camera 134 (or other image capture device) are coupled to the top panel 130, such as Figure 1 As shown. Alternatively, in some examples, at least one of the communication device 132 and / or the camera 134 is coupled to other parts of the paving machine 100. The communication device 132 is capable of determining the location of the paving machine 100 and includes and / or comprises components of a global positioning system (GPS). For example, the communication device 132 includes a GPS receiver, transmitter, transceiver and / or other such device, and the communication device 132 communicates with one or more GPS satellites (not shown) to determine the location of the paving machine 100 continuously, substantially continuously or at various time intervals. Alternatively or additionally, the communication device 132 communicates with a ground-based positioning station, a virtual reference station (VRS), a global navigation satellite system or other measurement coordinate system to determine the location of the paving machine 100. In examples, the communication device 132 also enables the paving machine 100 to communicate with one or more other machines, and / or with one or more remote servers, processors or control systems located away from the work site where the paving machine 100 is used, such as the computing device 204 including the pad defect ID component 214. In some examples, the camera 134 is a digital camera configured to record and / or transmit digital images and / or videos of the mat 120, paved surface 122, and / or worksite in real time. In some cases, the camera 134 includes an infrared sensor, thermal camera, or other similar device configured to record and / or transmit thermal images of the mat 120, paved surface 122, and / or worksite in real time. In an example, the camera 134 provides image data via the network 206 (e.g., via the controller 154) to the mat defect ID component 214, which uses the image data to identify defects of the mat 120 based on the image data.

[0026] like Figure 1 As shown, the paving machine 100 also includes one or more temperature sensors 136, 138, 140. One or more such temperature sensors 136, 138, 140 may include mechanical, electrical, electromechanical, electronic, infrared, or any other type of temperature sensor known in the art. In some examples, one or more such temperature sensors 136, 138, 140 include an air purge device or other such device (not shown) to prevent debris from forming on the temperature sensor. Such an exemplary air purge device receives clean air from an air source storing pressurized air to prevent any dirt, debris, etc. that may adhere to the temperature sensor.

[0027] In some examples, temperature sensor 136 includes a first temperature sensor coupled to paving machine 100 proximate auger assembly 114, and temperature sensor 136 is configured to sense, measure, detect, and / or otherwise determine the temperature of paving material 110 at auger assembly 114. Figure 1 As shown, the paver 100 includes a first side 142 (e.g., right side) and a second side 144 (e.g., left side) opposite the first side 142. Similarly, in such an example, the auger assembly 114 includes a first portion 146 disposed on the first side 142 of the paver 100 and a second portion 148 disposed on the second side 144 of the paver 100 (e.g., left side). Figure 1 14 and 15). In such examples, a first portion 146 of the auger assembly 114 includes a first main auger extending laterally from near the center of the auger assembly 114. The first portion 146 of the auger assembly 114 also includes a first auger extension coupled to the first main auger via a first auger bearing or other coupling component. Additionally, in these examples, a second portion 148 of the auger assembly 114 includes a second main auger opposite the first main auger, the second main auger extending laterally from near the center of the auger assembly 114. The second portion 148 of the auger assembly 114 also includes a second auger extension coupled to the second main auger via a second auger bearing or other coupling component. In such examples, the first main auger is substantially coaxially aligned with the second main auger to help deposit paving material 110 evenly on the paving surface 122 and / or substantially across the entire width of the paving machine 100.

[0028] In an example, the temperature sensor 136 is positioned near the first portion 146 of the auger assembly 114 and is configured to determine the temperature of the paving material 110 at the first portion 146 of the auger assembly 114. In addition, at least one additional temperature sensor (not shown) is positioned near the second portion 148 of the auger assembly 114 on the second side 144 of the paving machine 100. In such an example, the at least one additional temperature sensor is configured to determine the temperature of the paving material 110 at the second portion 148 of the auger assembly 114. In such an example, the temperature sensor 136 is positioned and / or otherwise configured to determine the temperature of the paving material 110 at and / or near the first auger bearing described above, or at other locations associated with the first portion 146 of the auger assembly 114. Similarly, in such an example, the at least one additional temperature sensor is positioned and / or otherwise configured to determine the temperature of the paving material 110 at and / or near the second auger bearing described above, or at other locations associated with the second portion 148 of the auger assembly 114. For example, temperature sensor 136 and / or additional temperature sensors include infrared sensors configured to sense, measure, and / or detect infrared radiation emitted by paving material 110 as paving material 110 passes through first and second auger bearings and / or just before paving material 110 is processed by screed portion 118. Temperature sensor 136 generates a signal including information indicative of the temperature of paving material 110 at first portion 146 of auger assembly 114. Likewise, the additional temperature sensors described above generate a signal including information indicative of the temperature of paving material 110 at second portion 148 of auger assembly 114. In some examples, temperature sensor 136 provides a signal including information indicative of the temperature of paving material 110 to pad defect ID component 214 (e.g., via controller 154), which uses the information included in the signal to identify a defect of pad 120.

[0029] Continue to refer Figure 1, temperature sensor 138 is positioned on tractor frame 106, hopper 108, and / or any other location on paving machine 100 that facilitates determining the temperature of paved surface 122. For example, temperature sensor 138 is positioned on tractor frame 106 near ground engaging element 104, and / or any other location that facilitates determining the ground temperature. Temperature sensor 138 is configured to generate a signal including information indicating the ground temperature. Additionally, temperature sensor 140 is positioned on top plate 130, tractor section 102, and / or any other location on paving machine 100 that facilitates determining the ambient temperature of the worksite. Temperature sensor 140 is also configured to determine the ambient pressure and / or other parameters at the worksite. In such an example, temperature sensor 140 generates a signal including information indicating an ambient condition such as ambient temperature, pressure, etc. The temperature sensor 138 and / or the temperature sensor 140 provide corresponding signals including information indicative of the temperature of the paved surface 122 or the ambient temperature of the worksite to (e.g., via the controller 154) the mat defect ID component 214, which uses the information included in the signals to identify defects in the mat 120 and / or generate instructions as described in more detail below.

[0030] In some examples, the paving machine 100 includes a lidar sensor 150 and / or a radar sensor 152. Although the lidar sensor 150 and / or the radar sensor 152 are depicted on the top plate 130, the lidar sensor 150 and / or the radar sensor 152 may be positioned anywhere on the paving machine 100, or positioned at another location near the paving machine 100 where the sensor captures sensor data of the mat 120. Additionally, although one lidar sensor 150 and one radar sensor 152 are shown, any number of lidar sensors and / or radar sensors may be incorporated into and / or near the paving machine 100. In an example, the lidar sensor 150 measures the distance to a target (e.g., the mat 120) by illuminating the target with a laser and measuring the reflected light with the sensor. The lidar sensor 150 uses the difference in laser return time and wavelength to make a digital 3D representation of the target. For example, the lidar sensor 150 generates a signal including information indicating a 3D representation of the target. Additionally, in some examples, radar sensor 152 uses radio waves to determine the range, angle, or speed of a target (e.g., mat 120). Radar sensor 152 generates a signal that includes information such as the range, angle, and / or speed of the target. It should be appreciated that paving machine 100 may include various other sensors to measure various other parameters related to paving machine 100, mat 120, and / or a worksite. Lidar sensor 150 and / or radar sensor 152 provide corresponding signals including information indicative of the worksite environment to (e.g., via controller 154) mat defect ID component 214, which uses the information included in the signal to identify a defect of mat 120 and / or generate instructions as described in more detail below.

[0031] The paving machine 100 also includes a controller 154 that communicates with the control interface 128, the communication device 132, the camera 134, the temperature sensors 136, 138, 140, the light radar sensor 150, the radar sensor 152, and / or other components of the paving machine 100. The controller 154 can be a single controller or multiple controllers that work together to perform various tasks. In an example, the controller 154 includes a single or multiple microprocessors, field programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other components configured to identify, calculate, and / or otherwise determine defects of the mat 120 based on one or more signals received from the camera 134, the temperature sensors 136, 138, 140, the light radar sensor 150, and / or the radar sensor 152. And / or radar sensor 152. Many commercially available microprocessors can be configured to perform the functions of the controller 154. Various known circuits may be associated with controller 154, including power supply circuitry, signal conditioning circuitry, actuator driver circuitry (i.e., circuitry that powers solenoids, motors, or piezoelectric actuators), and communication circuitry. In some examples, controller 154 is located on paving machine 100, while in other examples, controller 154 is located off-board and / or remotely from paving machine 100. The present invention is not limited in any way to the type of controller 154 or the location of controller 154 relative to paving machine 100.

[0032] Figure 2 1 is a block diagram schematically illustrating a control system 200 associated with a paving machine 100 according to an example of the present invention. In any examples described herein, control system 200 includes at least one of controller 154, control interface 128, communication device 132, camera 134, temperature sensor 136, an additional temperature sensor (not shown) described above with respect to second portion 148 of auger assembly 114, temperature sensor 138, temperature sensor 140, lidar sensor 150, radar sensor 152, and / or any other sensor or component of paving machine 100. In such examples, controller 154 is configured to receive corresponding signals from such components. For example, controller 154 receives one or more signals from communication device 132 that include information indicating the location of paving machine 100. As shown in FIG. Figure 2As shown, communication device 132 is connected to and / or in communication with one or more satellites 202 or other GPS components configured to assist communication device 132 in determining the location of paving machine 100. In some examples, such satellites 202 or other GPS components comprise components of control system 200. Controller 154, control interface 128, communication device 132, camera 134, temperature sensor 136, additional temperature sensors (not shown) described above with respect to second portion 148 of auger assembly 114, temperature sensor 138, temperature sensor 140, lidar sensor 150, radar sensor 152, and / or any other sensors or components of paving machine 100 provide information that may be used to determine defects in mat 120 and generate instructions provided to paving machine 100 to at least partially remedy the defects.

[0033] like Figure 2 As shown, the control system 200 may also include one or more additional components. For example, the control system 200 includes one or more remote servers, processors, or other such computing devices 204. Such computing devices 204 may include, for example, one or more servers, laptop computers, or other computers located at a paving material factory remote from the work site where the paving machine 100 is used. In such an example, the communication device 132 is connected to such computing devices 204 via a network 206 and / or otherwise communicates with such computing devices 204. The network 206 may be a local area network (LAN), a large network such as a wide area network (WAN), or a collection of networks such as the Internet. A protocol for network communication, such as TCP / IP, may be used to implement the network 206. Although embodiments are described herein as using a network such as the Internet, other distribution technologies for transmitting information via a memory card, flash memory, or other portable memory device may be implemented. The control system 200 may further include one or more electronic devices, such as a tablet computer, a mobile phone, a laptop computer, and / or other mobile device 208. Such mobile devices 208 may be located at the worksite, or alternatively, one or more such mobile devices 208 may be located at the paving material plant described above, or at another location remote from the worksite. In such examples, the communication device 132 is connected to and / or otherwise communicates with such mobile devices 208 via the network 206. In any of the examples described herein, sensor data, location information, paving material maps, and / or any other information received, processed, or generated by the controller 154 is provided to the computing device 204 and / or the mobile device 208 via the network 206.

[0034] In some examples, the controller 154 receives signals from one or more of the camera 134, the temperature sensor 136, the additional temperature sensor described above with respect to the second portion 148 of the auger assembly 114, the temperature sensor 138, the temperature sensor 140, the lidar sensor 150, and / or the radar sensor 152. For example, the controller 154 receives sensor data, such as images, videos, lidar data, radar data, infrared data, and / or other sensor data associated with the mat 120. Alternatively or additionally, the mobile device 208 includes a camera 210 (and / or any other sensor described herein) that generates sensor data, such as sensor data associated with the mat 120. For example, a worker operating the paving machine 100 may suspect that the mat 120 includes a defect and uses the camera 210 of the mobile device 208 to capture an image of the suspected defect in the mat 120. In some examples, the worker uses the interactive mat defect component 212 of the mobile device 208 to transmit the image to the computing device 204 via the network 206. In some cases, interactive pad defect component 212 is an application installed on mobile device 208 that enables a user, such as the worker just described, to submit sensor data associated with a suspected pad defect, compare the sensor data with sensor data known to be associated with a particular defect, receive instructions on how to control paving machine 100 to remedy the defect, etc., as described herein.

[0035] The controller 154 of the mobile device 208 and / or the interactive pad defect component 212 provide sensor data to the pad defect ID component 214. The pad defect ID component 214 is configured to identify a pad defect from the sensor data and map the pad defect to one or more instructions that will at least partially remedy the pad defect. For example, the pad defect ID component 214 includes a defect database 216 that includes images or other sensor data types known to illustrate pad defects and defect types of pad defects. In some cases, the defect database 216 includes sensor data showing different defects under different weather conditions, at different times of the day, at different ambient temperatures, with different paving material properties, etc. Additionally, in an example, the defect database 216 includes instructions associated with defects mapped to at least partially remedy the corresponding defects. In some cases, multiple different instructions are mapped to a single defect, wherein the multiple different instructions can be applied together to remedy the defect (e.g., increase the grade base and tighten the screed), and / or can be applied sequentially to remedy the defect (e.g., increase the grade base, and if the defect is not remedied, tighten the screed).

[0036] In the control system 200, the pad defect ID component 214 includes a feature detector 218 that is configured to compute an abstraction of image (or other sensor) information and make a local determination at an image point in the image as to whether the image point includes an image feature. For example, the feature detector 218 is configured to detect edges, corners, ridges, spots, or other image feature types. An edge may be characterized by a gradient magnitude of image brightness greater than a threshold between pixels of an image, where the pixels may or may not be close to each other. The characteristics of a corner are similar to an edge, but are further analyzed to determine rapid changes in edge direction and therefore have a two-dimensional structure. Ridges correspond to elongated features in an image represented by the main axis of symmetry of the captured feature, while edges generally involve the boundaries of an object. Spots provide information about regions of interest in an image by implementing image structure, which in some cases may provide higher accuracy than more point-like corners. For example, a detected spot may include an area in the image that is too smooth to be detected by a corner detector. In some examples, the pad defect ID component 214 selects a region of interest in an image (e.g., corresponding to the pad 120) on which the feature detector 218 will perform feature detection. However, in some cases, feature detector 218 detects features in the image before pad defect ID component 214 determines the location of the region of interest. Other feature detectors are also contemplated.

[0037] In some examples, the pad defect ID component 214 uses features identified by the feature detector 218 to locate the location of a potential pad defect of the pad 120 in the image. In the illustrative example, the pad defect ID component 214 detects an edge in the image where a difference in extender height and the height of the blade portion 118 of the paving machine 100 has resulted in the pad 120 having a different thickness. A relatively straight line may be represented in the image that defines a boundary between a first pad thickness formed by the blade portion 118 and a second pad thickness formed by the extender of the paving machine 100. The feature detector 218 detects the line, such as the edge, and the pad defect ID component 214 uses the location of the detected edge in the image to compare the suspected defect to known defects stored in the defect database 216.

[0038] For example, the pad defect ID component 214 includes a comparison component 220 configured to compare a potential pad defect in the sensor data (such as an image) with a known defect in the defect database 216. In an example, the comparison component 220 determines a value associated with a detected feature in the sensor data and compares the value with a known value (and / or a range of values) associated with a known defect in the defect database 216. In the case of using image data, the comparison component 220 determines whether the size of a first defect in the image data received from the camera 134 and / or the camera 210 is within a threshold size of a second defect, which is known to be a defect type indicated in the second image data stored in the defect database 216. The comparison component 220 relies on one or more of various characteristics of the defect and its associated value to determine the presence of the pad defect based on the sensor data. For example, the characteristics of the defect described herein include differences in material structure from an ideal pad 120, such as bumps, cracks, cracks, aggregate segregation, etc. Additionally, values ​​characterizing such exemplary characteristics include size values ​​(e.g., the length, width, height, depth, etc. of a suspected defect), temperature values, distance values, and / or angle values ​​(e.g., relative to the paving machine 100 or a particular portion of the paving machine 100, relative to a range of the mat 120, etc.), and the like. In some examples, the defect database 216 includes a range of values ​​associated with a characteristic that indicates the presence (and / or absence) of a defect in the mat 120. In an illustrative example, the defect database 216 includes a threshold size for large aggregate clusters, which, when present, indicates unacceptable segregation of aggregate particles in the mat 120. For example, if the threshold size in the defect database 216 is equal to 2 square feet, and if the size of the clusters of large aggregate particles on the mat 120 is greater than 2 square feet, then the comparison component 220 determines that a defect is present in the mat 120. Based at least in part on this determination, the comparison component 220 also determines that the defect corresponds to a defect type of unacceptable segregation of large and small aggregate particles.

[0039] The pad defect ID component 214 also includes an instruction component 222 that is configured to generate instructions associated with the defect based on a determination that the identified defect belongs to a particular defect type. In an example, the instruction component 222 generates instructions to identify a proposed machine activity that, when performed by the paver 100 (or other machine) on the paved surface, at least partially remedies the defect. For example, the instruction component 222 maps the defect type to one or more instructions stored in the defect database 216 that are known to at least partially remedy the defect type. The instruction component 222 provides the instructions to the mobile device 208 and / or the paver 100 via the network 206. In some cases, the instruction component 222 provides the instructions to the paver 100 such that the instructions cause the paver 100 to execute the instructions to remedy the defect without user intervention. For example, the instruction component 222 may receive an indication that the defect in the pad 120 is aggregate segregation, and thus generate instructions that cause the paver 100 to automatically adjust the height of the auger assembly 114 without user intervention to remedy the aggregate segregation. In these examples, paving machine 100 is a semi-autonomous or fully autonomous machine that is configured to navigate an environment and / or perform paving tasks without operator input.

[0040] In some examples, the pad defect ID component 214 also includes one or more machine learning models 224 configured to perform tasks related to defect identification in an image, defect type identification, instruction determination, etc. For example, the feature detector 218 utilizes the machine learning model 224 (e.g., size-invariant feature transform (SIFT), histogram of oriented gradients (HOG), etc.) and then utilizes a support vector machine (SVM) to classify objects depicted in images received from the camera 210 and / or the camera 134. Alternatively or additionally, the feature detector 218 can utilize a machine learning model 224 that utilizes a deep learning method based on a convolutional neural network (CNN) to classify objects depicted in images received from the camera 210 and / or the camera 134. For example, the feature detector 218 inputs at least a portion of the images received from the camera 210 and / or the camera 134 into a machine learning model 224 that is trained to identify paving defects in the images, such as one of the machine learning models just described. The feature detector 218 receives the location of the defect in the image and provides the location of the defect in the image to the comparison component 220, which uses the location of the defect in the image to determine a defect type of the defect.

[0041] Alternatively or additionally, the instruction component 222 utilizes a machine learning model 224 to determine instructions to provide to the electronic device to remedy the defect. For example, the instruction component 222 can utilize a machine learning model 224, such as a regression algorithm (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimated scatter plot smoothing (LOESS), a decision tree algorithm (e.g., classification and regression trees CART), iterative dichotomizer 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stumps, conditional decision trees), or other machine learning algorithms to determine which remedial measures are applied to which defects, how to apply different remedial measures in order to resolve the defects, which remedial measures are applied in combination to resolve the defects, and the like.

[0042] The machine learning models 224 that can be used for the pad defect ID component should not be limited to those just described. For example, the machine learning model 224 can be part of a neural network, which is a biologically inspired algorithm that passes input data through a series of connected layers to generate an output. Each layer in the neural network can also include another neural network, or can include any number of layers (whether convolutional or not). As can be understood in the context of the present invention, a neural network can utilize machine learning, which can refer to a broad category of such algorithms in which outputs are generated based on learned parameters.

[0043] Although discussed in the context of neural networks, any type of machine learning can be used in accordance with the present invention. In addition to those algorithms just described, machine learning algorithms can include, but are not limited to, example-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, averaged single phase dependency estimation algorithm (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, back propagation, Hopfield network, radial basis function, etc.), Function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA)), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), hybrid discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA), ensemble algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Hybrid), Gradient Boosting Machine (GBM), Gradient Boosting Regression Tree (GBRT), Random Forest), SVM (Support Vector Machine), supervised learning, unsupervised learning, semi-supervised learning, etc. Other examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, etc.

[0044] Although generally described above with respect to a single instance of sensor data received from a sensor of the paving machine 100 or the mobile device 208, examples are contemplated in which the pad defect ID component 214 utilizes multiple instances of sensor data to identify a pad defect and / or generate instructions to remedy a pad defect. For example, the pad defect ID component 214 may receive sensor data from different types of sensors. In an illustrative example, the pad defect ID component 214 may receive image data indicative of the pad 120 from the camera 134 and temperature data indicative of the pad 120 from one or more of the temperature sensors 136, 138, 140. The pad defect ID component 214 utilizes both sensor data types to identify a defect of the pad 120, such as by comparing the value of the sensor data type with known values ​​of corresponding sensor data types stored in the defect database 216 associated with different pad defects by the comparison component 220. Additionally, in some examples, the instruction component 222 generates instructions to be provided to the electronic device to remedy the defect based on the different sensor data types. For example, instruction component 222 may generate instructions based on a determination that temperature sensor data associated with the screed plate is within an acceptable range, instructing paving machine 100 to change the speed of the auger rather than the temperature of the screed plate to remedy the defect.

[0045] Alternatively or additionally, the pad defect ID component 214 may rely on multiple instances of the same type of sensor data to identify a defect and / or generate instructions to remedy the defect. For example, the pad defect ID component 214 may receive multiple instances of image data, such as a first image from the mobile device 208 and a second image from the camera 134 associated with the paving machine 100, and / or multiple images from different cameras mounted on the paving machine 100. In some cases, the first image data is generated by a first image capture device, and the second image data is generated by a second image capture device substantially simultaneously with the first image data. In other words, multiple instances of sensor data, such as images, are captured substantially simultaneously, but with different perspectives of the pad 120. For example, the pad defect ID component 214 uses additional perspectives of the pad 120 to identify pad defects that may not be present in a single image of the pad, and / or to refine the location of features identified by the feature detector 218. In this way, the feature detector 218 identifies features in the additional images and the locations of features in the additional images. The comparison component 220 compares the suspected defects at the locations of the identified features in the plurality of images, and may increase the confidence in the defect type of the determined defect based on the plurality of images of the defect having different viewing angles.

[0046] In some examples, the pad defect ID component 214 solicits user input, for example, via the interactive pad defect component 212, to determine the defect type of the suspected pad defect. For example, the pad defect ID component 214 determines that the value associated with the defect is within a value range of two (or more) different defect types. The pad defect ID component 214 generates a notification based on the number of defect types that the value falls within the respective value ranges, the notification including a first image corresponding to a first defect type from the defect database 216, a second image corresponding to a second defect type from the defect database 216, and the like. Additionally, the notification includes a request for a user (e.g., a worker at a worksite of the paving machine 100) to select which image is more similar to the defect currently encountered by the paving machine 100. The pad defect ID component 214 provides the notification to the interactive pad defect component 212 of the mobile device 208 via the network 206, which in turn displays the notification including the image and the request to the user. The interactive pad defect component 212 enables the user to select which image is more similar to the defect encountered at the paving machine 100, for example, by a touch input on one of the images in the user interface of the mobile device 208. The interactive pad defect component 212 provides the selection received from the user to the pad defect ID component 214 via the network 206. Upon receiving a selection of an image associated with one of the defect types, the instruction component 222 generates instructions based on the selection of the defect type. The pad defect ID component 214 stores the selection of the particular defect type with the raw image or sensor data in the defect database 216, which can be used for subsequent pad defect identification, to train the machine learning model 224, to generate subsequent notifications as just described, and the like.

[0047] Furthermore, in some cases, the pad defect ID component 214 stores sensor data received from sensors of the paving machine 100 and / or the mobile device 208 in the defect database 216 in association with the identified pad defect and known defect types. In this way, the defect database 216 grows to accommodate sensor data received under different conditions, at different times of the day, using different paving materials, etc., so that the pad defect ID component 214 becomes more accurate in identifying pad defects and generating appropriate instructions to remedy the pad defects. For example, the pad defect ID component 214 determines that a first defect indicated in the first image data is of a particular defect type. The pad defect ID component 214 then uses the first image in a subsequent comparison of images of suspected pad defects at a later time. For example, the pad defect ID component 214 determines that the second image indicates a defect of the particular defect type based on the similarity of the second defect indicated by the first image to the first defect indicated by the first image.

[0048] Figure 3300 is a schematic diagram of a paving machine 100 depositing paving material 110 on a paved surface 122. In schematic diagram 300, mat 120 is free of defects, as indicated by the generally uniform appearance of mat 120. In particular, mat 120 includes a uniform texture from a left side 302 of mat 120 to a right side 304 of mat 120, without blemishes or texture differences. In some examples, mat defect ID component 214 stores sensor data, such as image data, temperature data, infrared data, etc., corresponding to mats 120 without defects in defect database 216. Comparison component 220 compares the sensor data stored in defect database 216 corresponding to mats 120 known to be free of defects with sensor data received from sensors of paving machine 100 and / or mobile device 208 to determine whether a subsequently laid mat is also free of defects. For example, mat defect ID component 214 compares the size of a crack in mat 120 indicated in the image data to a threshold size (e.g., 2 inches, 6 inches, 12 inches, etc.). If the crack is less than the threshold size, the pad defect ID component 214 determines that at least the portion of the pad 120 is not defective. If the pad defect ID component 214 concludes that the pad 120 is not defective, the pad defect ID component 214 prevents the generation of instructions and / or prevents the instructions from being provided to the electronic device, as described elsewhere herein.

[0049] Figure 4 is a schematic diagram of an image 400 showing a mat 120 having a defect 402. In this example, the defect 402 corresponds to segregation of large aggregate and small aggregate (e.g., stone) in the paving material 110. As described above, segregation may be caused by various reasons, such as the position of the feed sensor, the auger height, the auger speed (e.g., revolutions per minute (RPM), the conveyor speed, the reversal of the auger, and / or the position or orientation of the deflector, etc.

[0050] exist Figure 4In the example shown, the pad defect ID component 214 receives the image 400 (and / or other sensor data corresponding to the mat 120, as described above) and uses the feature detector 218 to detect, for example, spots corresponding to the location of the defect 402 in the image 400. The comparison component 220 compares the values ​​associated with the characteristics of the defect 402 with the known defects in the defect database 216 to identify that the defect 402 corresponds to segregation of aggregate. The instruction component 222 generates one or more instructions 404 that, when executed by the paving machine 100, at least partially remedy the defect 402. In this case, the instructions 404 include adjusting the height of the auger, adjusting the speed of the auger, adjusting the position of the feed sensor, adjusting the mounting distance of the feed sensor, etc. If more than one instruction should be executed in association with the paving machine 100 to remedy the defect 402, two or more of the instructions 404 are provided together by the pad defect ID component 214 to the paving machine 100 and / or the interactive pad defect component 212. Alternatively or additionally, in response to determining that the first instruction does not remedy the defect, the pad defect ID component 214 sequentially provides two or more of the instructions 404 to the paving machine 100 and / or the interactive pad defect component 212 . Figure 5 and 6 Examples of components of paving machine 100 that may be modified according to instructions 404 to remedy a defect are shown.

[0051] Figure 5 yes Figure 1 Schematic diagram 500 of a paving machine 100 and associated components of paving machine 100 depositing paving material onto paved surface 122. Generally, different mixes of paving material 110 react differently to adjustments in the height of auger assembly 114. Thus, it may be necessary to adjust the auger height up or down based on the type of mix of paving material 110 and the appearance of mat 120 after paving begins.

[0052] The auger assembly 114 has a radius 504 extending from the perimeter of the auger assembly 114 to the center of the auger shaft 506 of the auger assembly 114. In some examples, the radius 504 of the auger assembly 114 is approximately equal to 8 inches. The height 508 corresponds to the height (e.g., depth) of the pad 120 before the pad 120 is compacted, and the height 508 extends from the uppermost surface of the paved surface 122 to the uppermost surface of the pad 120. In some examples, the height 508 is equal to between about 2 inches and about 4 inches. The height 510 corresponds to the distance between the uppermost surface of the pad 120 and the perimeter of the auger assembly 114 before the pad 120 is compacted. In some examples, the height 510 is equal to between about 2 inches and about 4 inches. In some examples, the height 512 of the auger assembly 114 is set equal to the sum of the radius 504, the height 508, and the height 510. Using the exemplary dimensions just described, the height 512 would be set between about 12 inches and about 16 inches. Other auger radii, auger heights, and / or other dimensions are also contemplated and may vary based on different auger sizes, different paving materials, etc.

[0053] In some cases, it may be necessary to adjust the height 512 of the auger assembly 114 after paving has begun. For example, if the height 512 of the auger assembly 114 is set below a specified ratio (e.g., 2:1) based on the material type and properties of the auger assembly 114, segregation of the pad 120 as shown by the defect 402 may result. The defect 402 may be more severe when the paving material 110 includes a larger stone mixture. In some examples, the instructions 404 instruct the paving machine 100 (and / or an operator of the paving machine 100 via the interactive pad defect feature 212 of the mobile device 208) to travel a distance at least equal to the full length of the paving machine 100 before evaluating whether the defect 402 has been resolved.

[0054] Figure 66 is a schematic diagram 600 of a feed sensor 602 included on paving machine 100 to determine the distance to paving material 110. Feed sensor 602 may be a mechanical paddle type, an acoustic wave sensor, or other type of sensor. In an example, feed sensor 602 generates a signal indicating the distance of feed sensor 602 from paving material 110 at the outboard end of auger 604, where paving material 110 is generally moving in a steady pattern. Feed sensor 602 should be positioned so that feed sensor 602 senses a moving pile of paving material 110, for example, approximately 18 inches (46 cm) from the last section of auger 604. In some examples, if feed sensor 602 is a paddle type sensor, the paddle arm of feed sensor 602 should be positioned at a distance of 18 inches at an approximately 45 degree angle from paving material 110. If feed sensor 602 is positioned too close relative to auger 604 (e.g., less than a threshold amount based on the composition of paving material 110), the signal generated by feed sensor 602 will be affected by the "wave" effect of paving material 110 from auger 604, and the operation of paving machine 100 will be erratic. If feed sensor 602 is too far from auger 604 (e.g., greater than a threshold amount based on the composition of paving material 110), the signal generated by feed sensor 602 may cause the feed system to overfill the auger chamber. Unstable operation of paving machine 100 or overfilling of the auger chamber may cause segregation as shown by defect 402.

[0055] In the example where feed sensor 602 is an acoustic wave sensor, when feed sensor 602 is aimed perpendicular to the surface of paving material 110, feed sensor 602 generates acoustic wave pulses 606 that propagate to paving material 110 and reflect back to feed sensor 602. Feed sensor 602 measures the time it takes for acoustic wave pulses 606 to travel to paving material 110 and back, and controller 154 calculates the distance from feed sensor 602 to paving material 110 based on the measured time. In the example where the distance between feed sensor 602 and paving material 110 that will prevent segregation is 18 inches (46 cm), when feed sensor 602 is an acoustic wave sensor, the operating range of feed sensor 602 may be from less than 12 inches (30 cm), causing controller 154 to cause the feed of paving material 110 to stop completely, to 32 inches (81 cm), causing controller 154 to cause the feed of paving material 110 to open completely. Controller 154 adjusts the flow of paving material 110 based on the sensed distance between 12 inches and 32 inches. Thus, the default distance from feed sensor 602 to paving material 110 is 18 inches, which results in an easily controlled mix height at the end of auger 604.

[0056] However, in some cases, feed sensor 602 may become misaligned with the paving material, which causes pulses 608 to bounce off feed sensor 602 instead of returning directly to feed sensor 602. When feed sensor 602 does not sense returning pulses 608, feed sensor 602 may operate erratically or not at all. As a result, the amount of paving material 110 being delivered may have an incorrect consistency and / or may cause mat 120 to be too thick or too thin, resulting in the segregation represented by defect 402. Therefore, instructions 404 include adjusting the feed sensor position and / or adjusting the feed sensor mounting distance to address the issues just described that may be associated with feed sensor 602 and cause the segregation represented by defect 402.

[0057] Figure 7 is another schematic representation of an image 700 of a pad 120 having a defect 702, in this case short waves / ripples in the pad 120 across the width of the pad 120. Ripples in the pad 120 as shown in image 700 may be caused by a variety of reasons, such as the screed plate slipping at a high angle of attack, the screed plate slipping on the lift cylinders, the screed plate being loose, and / or improper grade control settings, etc.

[0058] The pad defect ID component 214 receives the image 700 (and / or other sensor data corresponding to the pad 120, as described above) and uses the feature detector 218 to detect, for example, ridges and / or edges corresponding to the location of the defect 702 in the image 700. The comparison component 220 compares the values ​​associated with the characteristics of the defect 702 with the known defects in the defect database 216 to identify that the defect 702 corresponds to a corrugation in the pad 120. The instruction component 222 generates one or more instructions 704 that, when executed by the paving machine 100, at least partially remedy the defect 702. In this case, the instructions 704 include adjusting the composition of the mixture of paving material 110, adjusting the grade, inspecting the components of the screed portion 118, etc. If more than one instruction should be executed in association with the paving machine 100 to remedy the defect 702, two or more of the instructions 704 are provided together by the pad defect ID component 214 to the paving machine 100 and / or the interactive pad defect component 212. Alternatively or additionally, in response to determining that the first instruction does not remedy the defect, the pad defect ID component 214 provides two or more instructions 704 in succession to the paving machine 100 and / or the interactive pad defect component 212 .

[0059] Figure 88 is another schematic diagram of an image 800 showing a mat 120 having a defect 802. In this example, the defect 802 corresponds to an uneven surface texture of the mat 120. In some cases, an uneven surface texture of the mat 120 as shown in the image 800, where the unevenness occurs behind an extender 804 of the paving machine 100, may be caused by the extender having an angle of attack that is too low.

[0060] Pad defect ID component 214 receives image 800 (and / or other sensor data corresponding to pad 120, as described above) and uses feature detector 218 to detect, for example, edges corresponding to the location of defect 802 in image 800. Comparison component 220 compares values ​​associated with characteristics of defect 802 to known defects in defect database 216 to identify that defect 802 corresponds to an uneven pad texture that occurs behind an extender. Instruction component 222 generates one or more instructions 806 that, when executed by paving machine 100, at least partially remedy defect 802. In this case, instruction 806 includes adjusting the angle of extender 804, although other instructions described herein may also be included.

[0061] Fig. 9 is another schematic diagram of image 900 showing mat 120 having defect 902. In this example, defect 902 corresponds to a "torn spot" on the surface of mat 120. For example, paver 100 sets grade control on both sides of scraper portion 118 using an averaging ski on each side of scraper portion 118. The averaging ski controls scraper portion 118 so that scraper portion 118 fills low spots and peels off high spots of paving material 110 deposited on paved surface 122. In some cases, paver 100 reaches a location where the cold planer of paver 100 leaves a high spot on the grade, causing scraper portion 118 to lay mat 120 that is thinner than an optimal thickness. The ratio of the thickness of mat 120 to the size of aggregate in paving material 110 is below a threshold ratio, such as 2:1, causing mat 120 to have a torn appearance. Ideally, high spots on the grade are corrected before paving the pavement.

[0062] Pad defect ID component 214 receives image 900 (and / or other sensor data corresponding to pad 120, as described above) and uses feature detector 218 to detect, for example, edges and / or spots corresponding to the location of defect 902 in image 900. Comparison component 220 compares values ​​associated with characteristics of defect 902 to known defects in defect database 216 to identify that defect 902 corresponds to a torn pad 120. Instruction component 222 generates one or more instructions 904 that, when executed by paving machine 100, at least partially remedy defect 902. In this case, instructions 904 include locating irregularities in the grade of pad 120 that caused the high spot and the resulting tear, although other instructions described herein may also be included.

[0063] Fig.10 1 is an additional schematic diagram of an image 1000 showing a mat 120 with a defect 1002. In this example, defect 1002 corresponds to paving material 110 spilled on mat 120. In some cases, spilling paving material 110 on mat 120, as shown in image 1000, can result in high spots that cause mat 120 to tear after a spreader of paving machine 100 travels over the spillage. Spilled paving material 110 should be cleaned before the spreader passes over the spilled paving material 110 to prevent high spots.

[0064] Mat defect ID component 214 receives image 1000 (and / or other sensor data corresponding to mat 120, as described above) and uses feature detector 218 to detect, for example, spots corresponding to the location of defect 1002 in image 1000. Comparison component 220 compares values ​​associated with features of defect 1002 to known defects in defect database 216 to identify that defect 1002 corresponds to paving material 110 spread on mat 120. Instruction component 222 generates one or more instructions 1004 that, when executed by paving machine 100, at least partially remedy defect 1002. In this case, instructions 1004 include cleaning up spilled paving material 110, although other instructions described herein may also be included. In this case, instructions 1004 may be escalated using sound, haptics, etc. to send notifications to multiple workers to facilitate the time-sensitive nature of cleaning up spilled paving material 110 before the spreader reaches the spill location.

[0065] Fig.111 is a further schematic diagram of an image 1100 showing a pad 120 having a defect 1102. In this example, the defect 1102 corresponds to a pad 120 that is torn inside the path of the blade portion 118 of the paver 100. In some cases, the defect 1102 represented in the image 1100 may be caused by the blade extension being too low in height. Alternatively or additionally, the blade extension being too high may cause the line corresponding to the defect 1102 to align with the outer edge of the main blade.

[0066] The pad defect ID component 214 receives the image 1100 (and / or other sensor data corresponding to the pad 120, as described above) and uses the feature detector 218 to detect, for example, an edge corresponding to the location of the defect 1102 in the image 1100. The comparison component 220 compares the value associated with the characteristic of the defect 1102 with the known defects in the defect database 216 to identify that the defect 1102 corresponds to the pad 120 being torn in the path of the screed portion 118. The instruction component 222 generates one or more instructions 1104 that, when executed by the paving machine 100, at least partially remedy the defect 1102. In this case, the instructions 1104 include adjusting the screed extension height, adjusting the slope of the screed extension, etc. In some cases, two or more of the instructions 1104 are provided successively to the paving machine 100 and / or the interactive pad defect component 212 by the pad defect ID component 214 in response to a determination that the first instruction does not remedy the defect. In the illustrative example, the pad defect ID component 214 provides a first instruction to the interactive pad defect component 212, instructing the operator to lower the scraper extension until the defect 1102 disappears from the pad 120. If the defect 1102 recurs, the pad defect ID component 214 determines that the slope of the scraper extension is incorrect. Therefore, the pad defect ID component 214 provides a second instruction to the interactive pad defect component 212 to use the ramp extension switch to remove the defect 1102. In addition, in some cases, the pad defect ID component 214 also provides a third instruction and a second instruction to allow the operator to check the position of the extension ramp stop to prevent continuous problems with the ramp height.

[0067] Fig.12 1 is another schematic diagram showing an image 1200 of a pad 120 having a defect 1202. In this example, defect 1202 corresponds to an impact caused when a haul truck crashes into paving machine 100, causing blade portion 118 to gouge out pad 120.

[0068] Pad defect ID component 214 receives image 1200 (and / or other sensor data corresponding to pad 120, as described above) and uses feature detector 218 to detect, for example, edges and / or ridges corresponding to the location of defect 1202 in image 1200. Comparison component 220 compares values ​​associated with characteristics of defect 1202 to known defects in defect database 216 to identify that defect 1202 corresponds to a gouge formed by blade portion 118 when a haul truck struck paving machine 100 while depositing paving material 110. Instruction component 222 generates one or more instructions 1204 that, when executed by paving machine 100, at least partially remedy defect 1202. In this case, instructions 1204 include instructing an operator of the haul truck to stop near paving machine 100, instructing an operator of paving machine 100 to move paving machine 100 to the haul truck to unload paving material 110, and the like. In this case, instructions 1204 are sent substantially simultaneously to an electronic device associated with the operator of the haul truck and to an electronic device associated with the operator of paving machine 100 to ensure that both operators know what to do to remedy defect 1202 .

[0069] Fig.13 is a flow chart depicting a method 1300 of generating instructions for remediating an identified paving mat defect in accordance with an example of the present invention. The exemplary method 1300 is illustrated as a collection of steps in a logical flow chart that represents operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the steps represent computer-executable instructions stored in a memory. When the instructions are executed by, for example, the controller 154, the instructions generally cause the controller 154, various components of the control system 200, and / or the paving machine 100 to perform the operations described. Such computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described steps may be combined in any order and / or in parallel to implement the process. For purposes of discussion, and unless otherwise noted, the method 1300 is referred to herein. Figure 1 Paving Machine 100 and Figure 2 The control system 200 is described.

[0070] At 1302, pad defect ID component 214 receives sensor data, such as image data, representing a paved surface from paving machine 100 and / or mobile device 208, for example, via network 206. Although method 1300 generally refers to image data, controller 154 may alternatively or additionally receive other data types, such as video, lidar data, radar data, infrared data, etc., from camera 134 and / or camera 210, lidar sensor 150, radar sensor 152, temperature sensors 136, 138, 140, etc.

[0071] At 1304, the pad defect ID component 214 identifies a defect associated with the paved surface. For example, the feature detector 218 computes an abstraction of the image data and makes a local decision at an image point in the image data as to whether the image point includes an image feature. The feature detector 218 detects features such as edges, ridges, spots, or other feature types. Additionally, the pad defect ID component 214 uses the features identified by the feature detector 218 to locate the location of a potential pad defect of the pad 120 within the image.

[0072] At 1306, the pad defect ID component 214 determines that the defect belongs to a defect type. In some examples, the comparison component 220 compares the potential pad defect in the image data with known defects in the defect database 216. In an example, the comparison component 220 determines a value associated with a feature detected in the image data based in part on the feature detected by the feature detector 218. The comparison component 220 compares the value with a known value (and / or a range of values) associated with a known defect in the defect database 216. The comparison component 220 relies on one or more of various characteristics of the defect and its associated values ​​to determine the presence of the pad defect from the sensor data, such as a size value (e.g., the length, width, height, depth, etc. of the suspected defect), a temperature value, a distance value, and / or an angle value (e.g., relative to the paving machine 100 or a specific portion of the paving machine 100, relative to the range of the pad 120, etc.). In some examples, the defect database 216 includes a range of values ​​associated with the characteristics that indicate the presence (and / or absence) of the defect in the pad 120. For example, defect database 216 includes a threshold size of 2 square feet for large aggregate consolidation in mat 120. Comparison component 220 compares the suspected aggregate segregation defect in the image data to the threshold size stored in defect database 216 to determine whether mat 120 includes an aggregate segregation defect.

[0073] At 1308, the pad defect ID component 214 generates instructions associated with the defect that, when executed by a machine such as the paving machine 100, at least partially remedy the defect. For example, the instruction component 222 maps the defect type to one or more instructions stored in the defect database 216 that are known to at least partially remedy the defect type. In some cases, the instruction component 222 determines multiple instructions to be executed together (e.g., substantially simultaneously) or in parallel by one or more machines that at least partially remedy the defect. The instruction component 222 determines which machines to provide the instructions to, such as the paving machine 100, the mobile device 208, the haul truck, or other machine, based on how the instructions are to be executed and the machines mapped to the instructions in the defect database 216. For example, the instruction 1004 is provided to the mobile device 208 to instruct a worker to clean up spilled paving material 110, while the instruction 806 is provided directly to the paving machine 100 to automatically adjust the extender angle without user intervention.

[0074] At 1310, the pad defect ID component 214 determines whether the defect is remedied. For example, as described above, the pad defect ID component 214 compares the sensor data received after providing the instructions to the paving machine 100 and / or the mobile device 208 with the sensor data corresponding to the non-defective paving mats and the paving mats with suspected defects in the defect database 216. If the pad defect ID component 214 determines that the defect is remedied (e.g., "yes" at 1310), the pad defect ID component 214 continues to receive image data (or other sensor data) from the paving machine 100 and / or the mobile device 208 and continues to monitor the defect at 1312. If the pad defect ID component 214 determines that the defect is not remedied (e.g., "no" at 1310), the method 1300 returns to operation 1308. For example, the pad defect ID component 214 generates instructions different from the original instructions to remedy the defect, which instructions include different instructions to at least partially remedy the defect. Similar to the discussion above, the pad defect ID component 214 provides various instructions to an electronic device such as the paving machine 100 and / or the mobile device 208 .

[0075] Industrial Applicability

[0076] The present invention provides systems and methods for correcting defects associated with a mat 120 of paving material 110 formed at a worksite. Such systems and methods can be used to achieve better paving and compaction performance by remediating defects earlier and more accurately. Additionally, such systems and methods can be used to improve the efficiency of remediating defects by not having to rely on paving crews who are inexperienced with the various defects that may occur and how to resolve such defects. As described above with respect to Figure 1-13As noted, an exemplary method 1300 for correcting a paving mat defect may include receiving, with a controller 154, image data indicative of a paved surface 122, and identifying, with the controller 154 and based at least in part on the image data, a defect associated with the paved surface 122, the defect having a characteristic characterized by a value. Such exemplary method 1300 also includes: determining, with the controller 154, that the value is within a range of values ​​corresponding to a defect type; and determining, based at least in part on the value being within the range of values, that the defect belongs to the defect type. Such exemplary method 1300 further includes generating instructions 404 associated with the defect based at least in part on determining that the defect belongs to the defect type. In the exemplary method 1300, the instructions 404 identify a proposed machine activity that, when performed by the paving machine 100 on the paved surface 122, at least partially remedies the defect. Such exemplary method 1300 further includes providing the instructions 404 to an electronic device via the network 206.

[0077] By comparing the values ​​associated with suspected paving mat defects determined based on sensor data (e.g., image data) with known values ​​of different defect types, defects can be remedied more accurately and effectively than in the prior art. For example, the pad defect ID component 214 can provide instructions 404 to an electronic device associated with an operator of the paving machine 100, which instructs the operator how to change the settings of the paving machine 100 to remedy the detected defect. Alternatively or additionally, the pad defect ID component 214 can provide instructions 404 directly to the paving machine 100 to execute the instructions 404 to remedy the defect without user intervention, thereby allowing the paving machine 100 to autonomously correct the defect while the paving project is in progress. Such information can also be used by paving material manufacturers to more accurately manage paving material components and / or optimize the scheduling of transport truck deliveries based on the detected defect type and the estimated time to remedy the defect. Therefore, the exemplary systems and methods described above can provide considerable cost savings and can reduce the time and labor required to perform various paving activities on the site.

[0078] Although various aspects of the present invention have been specifically shown and described with reference to the above embodiments, it will be understood by those skilled in the art that various additional embodiments may be conceived by modifying the disclosed machines, systems and methods without departing from the spirit and scope of the disclosure. These embodiments should be understood to fall within the scope of the present invention as determined based on the claims and any equivalents thereof.

Claims

1. A method comprising: receiving, with a controller, image data indicative of a paved surface; identifying, with the controller and based at least in part on the image data, a defect associated with the paved surface, the defect having a characteristic characterized by a value; determining, with the controller, that the value is within a range of values ​​corresponding to a defect type; determining, with the controller and based at least in part on the value being within the range of values, that the defect is of the defect type; generating instructions associated with the defect and based at least in part on determining that the defect is of the defect type, wherein the instructions identify a proposed machine activity that, when performed by a machine on the paved surface, at least partially remedies the defect; and The instruction is provided to the electronic device via a network.

2. The method according to claim 1, wherein: The image data is received from an image capture device associated with one or more of the mobile electronic device or the paving machine.

3. The method according to claim 1, wherein the image data is first image data, the method further comprising: receiving second image data indicative of the paved surface; as well as identifying a defect associated with the paved surface based at least in part on the second image data, Wherein determining that the defect belongs to the defect type includes identifying the defect in both the first image data and the second image data.

4. The method according to claim 3, wherein: The first image data is generated by a first image capture device, and the second image data is generated by a second image capture device substantially simultaneously with the first image data.

5. The method of claim 1 , wherein the controller comprises a feature detector configured to identify features including one or more of edges, corners, or ridges from the image data, and Wherein identifying the defect comprises comparing a size of the feature to a known size associated with the defect type.

6. The method of claim 1, wherein the defect type is a first defect type and the value range is a first value range, the method further comprising: determining that the value is within a second range of values ​​corresponding to a second defect type; generating a notification including a first image associated with the first defect type, a second image associated with the second defect type, and a request for a user to select whether a defect associated with the paved surface is more similar to the first defect type or the second defect type; providing the notification to the electronic device; as well as receiving, from the electronic device, a user selection selecting the first defect type as being more similar to the defect than the second defect type; Wherein generating instructions associated with the defect is further based on receiving a user selection of the first defect type.

7. A method according to claim 1, wherein the image data is first image data and the defect belongs to a first defect, and wherein determining that the first defect belongs to the defect type is also based on determining whether a size of the first defect indicated in the first image data is within a threshold size of a second defect known to belong to the defect type and indicated in second image data.

8. The method according to claim 1, wherein: The electronic device controls a paving machine, and wherein the instructions cause the paving machine to remedy the defect without user intervention.

9. A system comprising: Sensing device; as well as a system controller in communication with a controller of the sensing device, the system controller being configured to: receiving sensor data indicative of a paved surface from the sensing device; identifying a defect associated with the paved surface based at least in part on the sensor data, the defect having a characteristic characterized by a value; determining that the value is within a range of values ​​corresponding to the defect type; determining that the defect belongs to the defect type based at least in part on the value being within the range of values; generating instructions associated with the defect and based at least in part on determining that the defect is of the defect type, wherein the instructions identify a proposed machine activity that, when performed by a machine on the paved surface, at least partially remedies the defect; and The instruction is provided to the electronic device via a network.

10. The system of claim 9, wherein the sensing device comprises an image capture device, a video capture device, a lidar sensor, a radar sensor, or an infrared sensor.

Images