Method of using a plurality of imaging devices and culture assessment system

Description

/ 29 METHOD OF USING A MULTIPLE IMAGING DEVICES AND A CROP EVALUATION SYSTEM DESCRIPTION FIELD

[001] This description refers generally to agricultural sprayers and other agricultural applicators that apply substances to a field. More specifically, the description is directed to the mechanical visualization of plants for controlling an agricultural sprayer or other agricultural applicator. FUNDAMENTALS

[002] An agricultural application machine is configured to apply an agricultural substance (liquid or dry forms) to a field. An example of an agricultural application machine is an agricultural sprayer or sprayer. An agricultural sprayer, for example, typically includes a tank or reservoir containing a substance to be sprayed onto an agricultural field. These systems typically include a supply line or conduit mounted on a collapsible, articulated, or retractable and extendable boom. The supply line is coupled to one or more spray nozzles mounted along the boom. Each spray nozzle is configured to receive the substance and direct it to a crop or field during application.As the sprayer moves across the field, the lance is moved into a deployed position, and the substance is pumped from the tank or reservoir, through the nozzles, so that the material is selectively applied (e.g., sprayed on) parts of the field.

[003] Instead of relying solely on an operator to determine when and where to direct the substance to be sprayed, automated systems have been developed to control the sprayer. For example, a machine display unit focused on areas below the spray nozzles can detect the presence of plants in the areas, determine the types of plants, and determine whether to apply a substance to them. Petition 870250072191, dated 08 / 15 / 2025, p. 13 / 55 / ​​​​29 plants. For example, if the substance is a herbicide, the visualization system can detect the presence of a weed and activate spraying for the area where the weed is identified. If the substance is a beneficial material for a crop plant, the visualization system can detect the presence of crop plants, assess the health of the crop plants, and selectively apply the material to the crop plants in response to the assessment.

[004] Several image processing algorithms have been developed to identify plant growth. Many of these algorithms use machine learning techniques to determine a plant type (e.g., a desired crop plant or a weed or other undesirable plant). These algorithms can also determine the overall health of a crop plant by comparing images of the crop plant with known images of the crop plant representing various conditions of those plants. The algorithms can also be applied by extracting information (parameters) from the images and applying the extracted information to an image calibration set.

[005] The accuracy of image processing algorithms depends in part on the quality of the images provided to the algorithms. The quality of images produced by a machine display unit depends on the spatial resolution of the images (e.g., the number of megapixels in each image frame) and also on the spectral resolution. A machine display unit, such as a standard still image or video camera, typically has three channels of spectral data that are generated by an array of red sensors, an array of green sensors, and an array of blue sensors. Each sensor type has a peak response at a respective peak wavelength and responds to wavelengths in a wavelength range that extends below and above the wavelength of Petition 870250072191, dated 08 / 15 / 2025, page 14 / 55 / ​​29 peak wave. The three wavelength bands overlap. An image pixel for a specific location in an image has a color that is created by the combination of the intensities of light detected for the corresponding red sensor, the corresponding green sensor, and the corresponding blue sensor at that specific location. Thus, an image pixel can represent a color other than red, green, and blue, even though the camera has sensors for colors other than red, green, and blue. This type of camera can be called an RGB camera.

[006] A machine visualization unit can also be a multispectral camera or a hyperspectral camera that provides many more color detection channels. Instead of just the three channels of a conventional RGB camera, a multispectral camera or a hyperspectral camera can have 40 or more channels of spectral data. Spectral data includes sensor data tuned to specific color wavelengths, in addition to the conventional red, green, and blue peak wavelengths. Each color wavelength corresponds to a frequency of light, where a shorter wavelength corresponds to a higher frequency of light and a longer wavelength corresponds to a lower frequency of light. Furthermore, each pixel on each sensor has a narrower band, so the output of each pixel represents the intensity of a narrower range of frequencies (colors).Some multispectral and hyperspectral cameras may include spectral channels that extend to wavelengths longer than visual wavelengths (e.g., the near-infrared spectral range) and to wavelengths shorter than visual wavelengths (e.g., the ultraviolet spectral range). As used herein, the channels of a multispectral camera may have overlapping spectral ranges, and the channels of a hyperspectral camera may have non-overlapping spectral ranges.

[007] Although multispectral cameras and hyperspectral cameras Petition 870250072191, dated 08 / 15 / 2025, page 15 / 55 / ​​29, to increase the accuracy of plant growth identification and plant health analysis, these cameras are much more expensive than conventional three-channel RGB cameras. For example, in an agricultural sprayer with 32 spray nozzles to be individually controlled, the cost of providing a multispectral camera or a hyperspectral camera to image the plants near each nozzle would be prohibitively expensive compared to using an RGB camera near each nozzle. SUMMARY

[008] In view of the foregoing, there is a need for a system and method to generate images of plant growth in a multi-unit agricultural applicator, such as an agricultural sprayer, using low-cost RGB cameras while simultaneously achieving higher image generation accuracy. The present description describes a system and method that uses a multispectral camera or a hyperspectral camera and at least one RGB camera to obtain images of the same plant to generate a correlation coefficient or correlatable features between the low spectral resolution images and the multispectral resolution images or the hyperspectral resolution images. For example, the correlatable features may be one or more shapes, textures, or spectra. The correlation coefficient or correlatable features are applied to low spectral resolution images of other plants produced by other RGB cameras to generate hyperspectral resolution images of the other plants.

[009] One aspect of the modalities described here is a system and a method that determine a physical characteristic of plants positioned under respective spraying units. The physical characteristic of each plant is determined by analyzing a respective image of each plant using image generation devices. A first image of each plant is obtained using a low spectral resolution camera. A second Petition 870250072191, dated 08 / 15 / 2025, page 16 / 55 / ​​29: An image of at least one plant is obtained using a higher spectral resolution camera. A correlation coefficient or correlatable features are generated based on the spectral characteristics of the elements of the first image and the spectral characteristics of the elements of the second image of at least one plant. The correlation coefficient or correlatable features are applied to the elements of each first image of the other plants to produce a respective higher spectral resolution image of each of the other plants. The spectral characteristics of the second images of the plants are analyzed to determine physical characteristics of at least one plant and of the other plants.

[0010] Another aspect of the modalities described herein is a method of using a plurality of low spectral resolution image-generating devices to determine features of a plurality of plants. The method comprises generating a first image of a first plant, wherein the first image-generating device has a first spectral resolution. The method further comprises generating a second image of the first plant using a second image-generating device with a second spectral resolution, wherein the second spectral resolution is higher than the first spectral resolution. The method correlates the second image of the first plant with the first image of the first plant to generate a correlation coefficient or correlatable features by mapping the first image of the first plant to the second image of the first plant.The method generates a respective first image of at least one plant in a group of plants using a respective third imaging device for each respective plant in the group of plants, wherein each third imaging device has the first spectral resolution. The method applies the correlation coefficient or correlatable features to the respective image. Petition 870250072191, dated 08 / 15 / 2025, page 17 / 55 / ​​29 first image of at least one plant in the group of plants to produce a respective second image generated of at least one plant in the group of plants. The respective second image generated of at least one plant in the group of plants has the second spectral resolution. The method analyzes the spectral characteristics of the second image of the first plant to determine a respective physical characteristic of the first plant. The method analyzes the spectral characteristics of the respective second image generated of at least one plant in the group of plants to determine a respective physical characteristic of at least one plant in the group of plants.

[0011] In certain embodiments according to this aspect, the first image-generating device and the third image-generating device are RGB cameras with three spectral resolution channels. The three channels comprise a red channel, a green channel, and a blue channel. The second image-generating device is a hyperspectral camera with more than three spectral resolution channels. In certain embodiments, the second image-generating device has at least 40 spectral resolution channels.

[0012] In certain embodiments according to this aspect, the method selectively activates a first application mechanism positioned close to the first image generation device to apply a material to the first plant when the respective physical feature of the first plant has a first characteristic. The method selectively activates a respective second application mechanism positioned close to a respective third image generation device to apply the material to at least one plant in the group of plants when the respective physical feature of at least one plant in the group of plants has a second characteristic. In certain embodiments, the second characteristic is the same as the first characteristic.

[0013] In certain modalities according to this aspect, the first Petition 870250072191, dated 08 / 15 / 2025, p. 18 / 55 / ​​​​29 characteristic is a type of plant. In certain embodiments, the plant type is a weed, and the material applied to the first plant and the second plant is a herbicide.

[0014] In certain embodiments according to this aspect, the first plant is a crop plant, the first characteristic is a relative health of the first plant, and the material applied to the plant is beneficial to the health of the plant. In such embodiments, at least one plant in the group of plants is a crop plant, the second characteristic is a relative health of at least one plant in the group of plants, and the material applied to at least one plant in the group of plants is beneficial to the health of at least one plant in the group of plants.

[0015] Another aspect of the embodiments described herein is a crop evaluation system. The system comprises a first machine visualization unit of a first type that is oriented to use a first image acquisition technique to obtain a first image of the first plant. The first image of the first plant has a first spectral resolution. A machine visualization unit of a second type is oriented to use a second image acquisition technique to obtain at least one second image of the first plant with a second spectral resolution. The second spectral resolution is greater than the first spectral resolution. A second machine visualization unit of the first type is oriented to use the first image acquisition technique to obtain a first image of a second plant. The first image of the second plant has the first spectral resolution.The system includes a processing system configured to receive the first image of the first plant and the second image of the first plant. The processing system maps elements from the first image of the first plant to elements of the second image of the first plant to generate a correlation coefficient or correlatable features between the first image and the second image. Petition 870250072191, dated 08 / 15 / 2025, page 19 / 55 / ​​29 first plant at the first spectral resolution and the second image of the first plant at the second spectral resolution. The processing system is further configured to receive the first image of the second plant. The processing system applies the correlation coefficient or correlatable features to the elements of the first image of the second plant to produce a second generated image of the second plant. The processing system is further configured to analyze the spectral features of the second image of the first plant to determine at least one physical feature of the first plant; and to analyze the spectral features of the second generated image of the second plant to determine at least one physical feature of the second plant.

[0016] In certain embodiments, according to this aspect, the evaluation system further comprises a sprayable material source and at least one first spraying unit and one second spraying unit. The first and second spraying units are coupled to receive the sprayable material from the sprayable material source. The first spraying unit is positioned close to the first machine display unit of the first type and is positioned close to the first machine display unit of the second type. The second spraying unit is positioned close to the second machine display unit of the first type. The first spraying unit is controllable to selectively spray the sprayable material onto a first plant close to the first spraying unit in response to receiving a first command.The second spraying unit is controllable to selectively spray the sprayable material onto a second plant adjacent to the second spraying unit in response to receiving a second command. The processing system is further configured to selectively send the first command to the first spraying unit to activate it. Petition 870250072191, dated 08 / 15 / 2025, page 20 / 55 / ​​29 spraying unit in response to at least one determined physical characteristic of the first plant, and to selectively send the second command to the second spraying unit to activate the second spraying unit in response to at least one determined physical characteristic of the second plant.

[0017] In certain embodiments according to this aspect, at least one physical characteristic of the second plant is a plant type. In certain embodiments, the type of the second plant is an undesirable plant; and the material applied to the second plant is a herbicide.

[0018] In certain embodiments according to this aspect, at least one physical characteristic of the second plant is a relative health of the second plant; and the material applied to the second plant is a material beneficial to the health of the second plant.

[0019] Several objects, features and advantages of the embodiments presented here will be readily apparent to those skilled in the art after reading the following description when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 illustrates a pictorial illustration of an agricultural application system operating in a field, the agricultural application system including a structure supporting a plurality of applicator units.

[0021] Figure 2 illustrates an enlarged illustration of a first type of applicator / imaging unit of the agricultural application system of Figure 1, the first type of applicator / imaging unit including a spraying mechanism and including a low spectral resolution imager (e.g., a machine visualization unit, such as a camera) having a first spectral resolution.

[0022] Figure 3 illustrates a simplified spectrum of the low spectral resolution imager of the first type of applicator / imaging unit of Petition 870250072191, dated 08 / 15 / 2025, page 21 / 55 / ​​29 Figure 2.

[0023] Figure 4 illustrates an enlarged illustration of a second type of applicator / imaging unit of the agricultural application system of Figure 1, the second type of applicator / imaging unit including a spraying mechanism, a low spectral resolution imager having the first spectral resolution and a high spectral resolution imager having a second spectral resolution.

[0024] Figure 5 illustrates a simplified spectrum of the high spectral resolution imager of the second type of applicator / imaging unit of Figure 4.

[0025] Figure 6 is a simplified system block diagram showing interconnections between a control unit and the first type of imager / applicator units and the second type of imager / applicator unit of Figure 1.

[0026] Figure 7 illustrates a block diagram of the control unit in Figure 6.

[0027] Figure 8 is a flowchart of the operating method of the system in Figure 6. DETAILED DESCRIPTION

[0028] Figure 1 is a pictorial illustration of an agricultural application system 100 in a field 110 with multiple ridges 112 (illustrated in dashed lines) in parallel rows. The field is illustrated as having a plurality of plants 114. The agricultural application system is illustrated as an agricultural sprayer; however, the embodiments described here are easily adaptable to other agricultural application systems with multiple applicator devices that can be individually controlled.

[0029] The agricultural application system 100 includes a tractor 120 or other source of mobility and a removable applicator structure 122 attached to Petition 870250072191, dated 08 / 15 / 2025, page 22 / 55 / ​​29 tractor by means of an articulated mechanism 124. Although illustrated as a tractor and a removable applicator structure, the two devices can be combined as an integral self-propelled application system.

[0030] The tractor 120 supports a material source (e.g., a container) 130 of a material to be applied to the field 110. For example, the material to be applied may be a herbicide to be sprayed on an unwanted plant for weed control. The material may also be a fertilizer, another nutrient, an insecticide, or something similar to be sprayed on a desired plant. The material to be applied is transported from the source to the applicator system by a conduit 132.

[0031] The removable applicator structure 122 includes a support structure 140 that extends from the articulated mechanism 124 in two directions, so that the support structure extends over multiple rows of crops on both sides of the tractor 120. Although illustrated as a single unit, the support structure may include hinges (not shown) that allow all or part of the support structure to be raised for transport or for applying material only to the side of the tractor 120.

[0032] The support structure 140 supports a plurality of a first type of applicator / imager unit 150 and at least one second type of applicator / imager units. The applicator / imager units are spaced in the support structure by the approximate distance between the crests 112 of the field 110. In Figure 1, ten instances of the first type of applicator / imager unit are shown; however, additional instances of the first type of applicator / imager unit may be included for long versions of the support structure. For example, a specific embodiment may include 32 or more units of the first type of applicator / imager. Each unit of the first type of imager / applicator is aligned with a peak of one of the respective crests. The support structure also supports at least one second type Petition 870250072191, dated 08 / 15 / 2025, page 23 / 55 / ​​29 of imager / applicator unit 160, which is aligned with the peak of one of the ridges. In the illustrated embodiment, the second type of applicator / imaging unit is shown approximately in the middle of the support structure; however, the second type of applicator / imaging unit may also be located elsewhere on the support structure, in place of one of the imager / applicator units of the first type.

[0033] As shown in Figure 2, each of the first type of applicator / imager unit 150 comprises a controllable spraying mechanism (applicator) 200 having a nozzle 202, which comprises the applicator portion of the unit. The nozzle is oriented to selectively spray material from the material source 130 (Figure 1) onto a spraying area 204 generally centered on the respective crest 112 underlying the nozzle.

[0034] As shown in Figure 2, each of the units of the first type of applicator / imager 150 also comprises an imager 210 of a first type. In the embodiment described herein, the imager of the first type is a machine visualization unit, such as a camera. The following references to a camera are intended to encompass other types of imagers and machine visualization units. The camera of the first type is described herein as an RGB camera and will be referred to as an RGB camera in the following description. Each RGB camera 210 has an associated lens 212. In the embodiment illustrated in Figure 2, the RGB camera is directed perpendicularly downwards towards the plant 114, generally below the camera. In other embodiments, the RGB camera may be directed forward, backward, or to one of the sides. For example, in one embodiment (not shown), the camera is directed forward approximately 44 degrees from perpendicular.

[0035] In the embodiment illustrated, the RGB 210 camera may comprise a conventional RGB camera that receives light by means of three Petition 870250072191, dated 08 / 15 / 2025, page 24 / 55 / ​​29 sets of sensors, wherein each set of sensors comprises an array of sensors that are optimized for light sensitivity in a specific range of wavelengths. For example, as illustrated by a spectrum 230 in Figure 3, a “blue” sensor is sensitive to light in a first range of 232 wavelengths from approximately 400 nanometers to approximately 525 nanometers, with a peak sensitivity around 475 nanometers. A “green” sensor is sensitive to light in a second range of 234 wavelengths from approximately 400 nanometers to approximately 620 nanometers, with a peak sensitivity around 540 nanometers. A "red" sensor is sensitive to light in one-third of the 236 wavelength range, from approximately 560 nanometers to approximately 675 nanometers, with a peak sensitivity around 590 nanometers.Each sensor array outputs a digital value for each sensor in the array, where the digital value of a sensor represents the intensity of light in the specific wavelength range incident on the sensor. The digital values ​​of the corresponding sensor locations in each array are combined to form a pixel value that has three channels of color information: a red channel, a green channel, and a blue channel. For example, each pixel value can be represented by 24 bits of data (3 bytes of data), with each 8-bit byte representing 256 intensity levels for each bandwidth. As illustrated in Figure 3, the bandwidths for the three colors overlap so that other colors in the spectral range from 400 nanometers to 700 nanometers are detectable as intensities by two or more color sensors in the sensor array. The RGB camera outputs data representing the intensities of red, green, and blue in each frame of the image.Can we identify a camera that can be used for RGB camera work?

[0036] As is well known in art, each RGB image created by Petition 870250072191, dated 08 / 15 / 2025, page 25 / 55 / ​​29 each RGB 210 camera is stored as digital data representing the intensity of red, blue-green, and blue for each pixel of the image. Thus, for example, the image from a 16-megapixel RGB camera would occupy 15,772,256 storage locations for a pixel matrix of 5,312 pixels by 2,988 pixels. Each storage location can occupy 24 bits (3 bytes) when each color intensity is represented by 8 bits (256 levels). Each memory location can occupy more bits for greater color differentiation. The pixel data from the cameras is output as described below.

[0037] The RGB camera 210 and lens 212 are adjusted to focus on a first image area 214, which generally covers the spray area 204. In the illustration, one of the plants 114 is shown within the spray area and the image area.

[0038] The preceding description of the first type of applicator / imager unit 150 is one of many different configurations that can be used to position the RGB camera 210 and the spraying mechanism 200 close to each other, so that the imager is directed to the area where the material is to be applied (e.g., sprayed). In other embodiments, the first type of camera and the spraying mechanism may be separate units that are mounted independently on the support frame 140.

[0039] Figure 4 illustrates the second type of applicator / imaging unit 160 in more detail. Similar to the first type of applicator / imaging unit 150 described earlier in Figure 2, the second type of applicator / imaging unit includes the spray mechanism 200 and the nozzle 202. The second type of applicator / imaging unit also includes an imager of the first type (RGB camera) 310 and an associated lens 312, which correspond to the RGB camera 210 and the lens 212 as described earlier for the first type of unit. Petition 870250072191, dated 08 / 15 / 2025, p. 26 / 55 / ​​​​29 of applicator / imager 150 of Figure 2. The RGB camera and associated lens in the second type of applicator / imager unit are adjusted to focus on a second image area 314 that has a shape and size generally corresponding to the first image area 214 of Figure 2.

[0040] Unlike the first type of applicator / imager unit 150 described earlier, the second type of applicator / imager unit 160 in Figure 4 also includes an imager of a second type 320. As described below, the camera of the second type is either a multispectral camera or a hyperspectral camera. As used herein, a multispectral camera has multiple channels (e.g., 40 channels) of overlapping spectral bands, and a hyperspectral camera has multiple channels (e.g., 40 channels) of non-overlapping spectral bands. The following description is directed to the camera of the second type, which is a hyperspectral camera; however, the description is applicable to the camera of the second type, which is a multispectral camera. The description also applies to the camera of the second type with more than 40 channels of spectral bands.The hyperspectral camera and associated lens 322 are adjusted to focus on a third image area 324, which overlaps at least partially with the second image area 314. In Figure 4, the third image area is shown as being slightly larger in area than the second image area; however, the third image area may also have the same area as the second image area or it may have a smaller area than the second image area. The third image area is also shown as having the same shape and orientation as the second image area. The images may be distorted and encompass different areas; however, conventional image processing can be used to reorient and resize one of the image areas relative to the other imager area, recognizing common image features in the images generated by the first camera type and the second camera type. Petition 870250072191, dated 08 / 15 / 2025, page 27 / 55 / ​​29 according to the type of applicator / imaging unit.

[0041] As discussed above with regard to the first type of applicator / imager unit 150, the second type of applicator / imager unit 160 can also be configured with the RGB camera 310, the hyperspectral camera 320 and the spray mechanism 200 installed independently on the support structure 140.

[0042] In the illustrated embodiment, the hyperspectral camera 320 in the second type of applicator / imager unit 160 has 40 or more spectral data channels covering a wavelength range from 400 nanometers (blue) to 700 nanometers (red). {Can we identify a camera that can be used as the second type of camera?} The spectral bandwidth for an example of the hyperspectral camera is illustrated as a spectrum 330 in Figure 5. As illustrated, the spectrum is divided into 40 wavelength bands between 400 nanometers and 700 nanometers, although more or fewer bands may be used. Each band has an overall bandwidth of approximately 7.5 nanometers and represents the sensitivity of a specific sensor, with the maximum sensor sensitivity being approximately in the center of the illustrated bandwidth.Each band can overlap an adjacent band by a small amount (for example, if the second type of camera is a multispectral camera); however, no overlap is illustrated in Figure 5 for the modality shown using a hyperspectral camera. Each pixel location in the second type of camera has 40 sensors, with each sensor sensitive to one of the 40 light bandwidths. For example, a sensor sensitive to a lower band of 332 wavelengths from approximately 400 nanometers to approximately 407.5 nanometers has a maximum sensitivity at approximately 403.75 nanometers, corresponding to violet light. A sensor sensitive to a medium band of 334 wavelengths from approximately 465 nanometers to... Petition 870250072191, dated 08 / 15 / 2025, page 28 / 55 / 29 approximately 472.5 nanometers has a maximum sensitivity of approximately 468.75 nanometers, corresponding to blue light. A sensor sensitive to an upper mid-range of 336 wavelengths from approximately 535 nanometers to approximately 542.5 nanometers has a maximum sensitivity of approximately 538.75 nanometers, corresponding to green light. A sensor sensitive to an upper 338 wavelength range from approximately 670 nanometers to approximately 677.5 nanometers has a peak sensitivity at approximately 673.75 nanometers, corresponding to red light. The wavelengths above may vary in different cameras and are used here for illustration purposes only. The second type of camera provides output data that represents the light intensity for each bandwidth at each pixel location.For example, assuming 256 intensity levels for each channel, which can be encoded as 8 bits of data, each pixel could require up to 10,240 bits (1,280 bytes) of raw data, resulting in an output of approximately 20.2 gigabytes of data from a 16-megapixel camera for each image frame. A suitable encoding technique can be used to reduce the amount of output data, since many of the sensors associated with the 40 channels will not receive light at their respective bandwidths in each image frame.

[0043] As illustrated in a simplified system block diagram 400 in Figure 6, the first ten types of applicator / imager units 150 and the second type of applicator / imager unit 160 communicate control and information bidirectionally to and from a control unit 410. Many different configurations can be used to implement the bidirectional communications. In the simplified version illustrated in Figure 6, the control unit includes a respective power cable 420 for each of the spraying mechanisms 200 within the first type of applicator / imager units and for the mechanism of Petition 870250072191, dated 08 / 15 / 2025, page 29 / 55 / ​​29 spraying within the second type of applicator / imager unit, such that the control unit supplies power to one of the spraying mechanisms to activate the respective mechanism and removes power to deactivate the respective mechanism. In an alternative configuration (not shown), a single power cable may be provided for multiple spraying mechanisms, and the control unit may send signals to control units (not shown) within the respective spraying mechanisms to cause the control unit to selectively connect electrical power to the power cable as instructed. For example, signals may be sent via a multiplexed bus, via dedicated low-current signal lines, or via wireless communications.The system described here is not limited to a specific communication configuration for controlling spraying mechanisms.

[0044] The ten RGB 210 cameras in the first type of applicator / imager unit 150 and the RGB 310 camera and the hyperspectral 320 camera of the second type of applicator / imager unit 160 also communicate with the control unit 410. The cameras can be connected directly to the control unit using a respective control and data bus 430, as illustrated in Figure 6, or using Ethernet cabling in a local area network (LAN) or similar configuration. In one embodiment, the local area network is a CAN (controller area network) bus. Alternatively, the cameras can communicate with the control unit via a wireless communication protocol.

[0045] In the selected configuration, each RGB camera 210 in the first type of applicator / imager unit 150 sends data to the control unit 410 which represents an RGB image of the field portion in the respective first image area 214. The RGB camera 310 in the second type of applicator / imager unit 160 sends data to the unit of Petition 870250072191, dated 08 / 15 / 2025, page 30 / 55 / ​​29 control representing an RGB image of the second image area 314. The hyperspectral camera 320 in the second type of applicator / imager unit sends data to the control unit representing a hyperspectral image of the third image area 324.

[0046] The control unit 410 is illustrated in more detail in Figure 7. The control unit includes a respective RGB image memory storage location 450 associated with each of the first ten types of applicator / imager units 150 of Figure 1. Each image memory storage location receives respective RGB pixel image data from a respective RGB camera 210 in the first type of applicator / imager unit via the respective control and data bus 430. The control unit includes another RGB image memory location 452 to receive RGB pixel image data from the RGB camera 310 in the second type of applicator / imager unit 160. The control unit includes a hyperspectral image memory location 454 to receive hyperspectral pixel image data from the hyperspectral camera 320 in the second type of applicator / imager unit via the respective control and data bus 430.

[0047] The hyperspectral pixel image data stored in hyperspectral image memory location 454 is applied as an input to a first hyperspectral image processing and interpretation unit 460. The first hyperspectral image processing and interpretation unit analyzes the hyperspectral pixel image data based on parameters provided to the unit to determine the type of plant growth, if any, in the image. For example, if the agricultural application system 100 (Figure 1) is currently being used to selectively apply herbicide to unwanted plants in a soybean field. The parameters applied to the first hyperspectral image processing and interpretation unit are used to Petition 870250072191, dated 08 / 15 / 2025, page 31 / 55 / ​​29 to identify a soybean plant. If a plant other than a soybean plant is identified, the first hyperspectral image processing and interpretation unit emits an application signal to a sprayer control unit 462, which sends an appropriate signal to the controllable spraying mechanism 200 in the second type of applicator / imager unit to selectively activate the spraying mechanism to apply the herbicide to the unwanted plant. Similarly, the first hyperspectral image processing and interpretation unit can receive parameters to identify a desired plant (e.g., a soybean plant), assess the plant's condition based on the parameters, and apply, for example, fertilizer, another nutrient, or an insecticide to the plant, emitting an application signal as discussed above.Image processing and evaluation to perform the aforementioned functions are well-known in relation to agricultural image generation and are not described in detail here. See, for example, “Global Hyperspectral Imaging Spectral Library of Agricultural Crops (GHISA); Study Area: Contiguous United States (CONUS); Algorithm Theoretical Basis Document (ATBD)”, August 2019, version 2.0, USGS, Sioux Falls, South Dakota. Due to the multiple bands (e.g., 40 bands) of spectral images provided by such cameras, the camera images can be processed using known algorithms to identify types of plant material (e.g., crops such as corn, rice, cotton, soybeans, and the like, as well as non-crops such as weeds). The algorithms are able to identify the condition of a photographed crop to detect diseases, water content, pests, nutrients, and the like.

[0048] The second type 320 hyperspectral camera in the second type of applicator / imager unit 160 is capable of providing hyperspectral images only of plants on the respective ridge 112 immediately below the position of the hyperspectral camera. Thus, the RGB cameras 210 in Petition 870250072191, dated 08 / 15 / 2025, page 32 / 55 / ​​29 first type of applicator / imager units 150 provide plant images below the first type of applicator / imager units. Although the cameras in the first type of applicator / imager units could also be hyperspectral cameras, hyperspectral cameras cost much more than conventional RGB cameras. The additional cost of providing hyperspectral cameras for each of the first ten types of applicator / imager units would be prohibitively expensive for a support structure with only 11 applicator / imager units 150,160, as illustrated. The cost difference would be much greater for a larger support structure 140 having, for example, 32 imager / applicator units. Thus, as described above, the first type of applicator / imager units uses much cheaper RGB cameras.However, the RGB pixel data from the RGB camera cannot be used to process and evaluate the plants captured by the RGB cameras.

[0049] Control unit 410 in Figure 6 includes additional processing elements that allow the control unit to use the RGB pixel data from the RGB cameras 210 to determine the characteristics of the plants located below the cameras. The control unit includes an RGB-to-hyperspectral mapping system (correlation unit) 500 that correlates the RGB image from the RGB camera 310 in Figure 4 to the hyperspectral image from the hyperspectral camera 320 in Figure 4. The mapping system receives the RGB pixel image data from the RGB memory location 452 which stores the image data generated by the RGB camera 310 in the second type of applicator / imager unit 160. The RGB pixel image data represent the second image area 314 in Figure 4.The mapping system also receives hyperspectral pixel data from hyperspectral memory location 454, which stores image data generated by the hyperspectral camera in the second type of applicator / imaging unit. This is the hyperspectral image data. Petition 870250072191, dated 08 / 15 / 2025, page 33 / 55 / ​​29 represent area 324 in Figure 4.

[0050] The 500 mapping system first unskews and resizes the two images relative to each other. Framing and resizing images to allow comparison of two images is well known in the field and is described in detail here. Because RGB image data comprises fewer memory locations than hyperspectral data, the RGB image data is modified relative to the fixed hyperspectral data. If the spatial pixel sizes of the images are the same (e.g., both images are 16 megapixels (5,312 pixels by 2,988 pixels)), the two images are ready for the next step.If, for example, an image has a different number of pixels (for example, 4 megapixels instead of 16 megapixels) or has a different aspect ratio, one of the images can be modified by interpolation or another known technique to conform the two images so that each pixel in one image represents the same image as the correspondingly located pixel in the other image.

[0051] Several techniques are known and can be used to create mapping data (e.g., correlation coefficient or correlatable features) to convert a hyperspectral image into an RGB image. In one form of image mapping, each possible combination of RGB pixels (e.g., an 8-bit R value; an 8-bit G value; and an 8-bit B value) is used as an index value for the map. As the pixels in the RGB image are evaluated (in each row in the sequence), the RGB combination at each pixel location is used as an index for the map. The hyperspectral value for the corresponding pixel location in the hyperspectral image is stored in a location corresponding to the index value. When the last row of the pixel image is complete, a partial map is created, including a hyperspectral value for each RGB combination found in the RGB image. Petition 870250072191, dated 08 / 15 / 2025, page 34 / 55 / ​​29 RGB camera 310 in the second type of applicator / imager unit 160. For unidentified RGB combinations in the RGB image from the RGB camera in the first type of applicator / imager unit, interpolation can be used to fill in the hyperspatial values ​​from the nearest identified RGB values. Interpolation can occur as the mapping is created, or as the mapping is used, as described below. If, during the mapping process, a specific RGB index value occurs at another pixel location in a pixel of the RGB image and the hyperspectral value at the corresponding location of the hyperspectral image is different from the hyperspectral value previously stored in the map, the two different hyperspectral values ​​can be reconciled using known techniques. Other mapping techniques can also be used.

[0052] The mapping data (correlation coefficient or correlatable features) of the mapping system 500 are provided as an input to a respective hyperspectral imager 520 associated with each of the RGB image memory storage locations 450 that store images from the RGB cameras 210 in the first type of applicator / imager unit 150. The RGB pixel data in each pixel in each row are read in order and used as an index for the mapping data in the respective hyperspectral imager. Each index value results in hyperspectral data being output by the hyperspectral imager. The hyperspectral data are stored in a respective hyperspectral memory 522 associated with a respective first type of applicator / imager unit 150.If the mapping information is not populated for a specific index value, interpolation can be used to generate hyperspectral data for the corresponding pixel in hyperspectral memory.

[0053] Outputs of each 522 hyperspectral memory for images from the RGB 210 cameras of the first type of applicator / imaging units Petition 870250072191, dated 08 / 15 / 2025, page 35 / 55 / ​​29 150 are provided as inputs to respective second hyperspectral image processing and interpretation units 530 which correspond to the first hyperspectral image processing and interpretation unit 460 described above. Each second hyperspectral image processing and interpretation unit receives the parameters received by the first hyperspectral image processing and interpretation unit and selectively generates a corresponding application signal for a respective sprayer control unit 532. Each sprayer control unit sends an appropriate signal to the controllable spraying mechanism 200 in the first type of applicator / imager unit to turn on the mechanism and selectively apply a herbicide to an unwanted plant or selectively spray a beneficial material to a desired plant.

[0054] The control unit 410 can be located in various locations relative to the tractor 120 and the removable applicator frame 122. For example, in one embodiment, the control unit is located on the tractor so that the control unit electronics benefit from the air-conditioned environment of the tractor.

[0055] The control unit 410 of Figures 6 and 7 operates according to flowchart 600 of Figure 8. In a first step 610 of the operation, the control unit sends instructions to the RGB camera 310 (the camera of the first type) in the second type of applicator / imager unit 160 to cause the RGB camera to create an image within the second image area 314 (Figure 4). The image may include an image of a plant 114. As discussed above, the RGB camera has a first spectral resolution as illustrated in Figure 3. The image information is communicated to the control unit via the control and data bus 430 or via the LAN (not shown), according to the communication configuration. Petition 870250072191, dated 08 / 15 / 2025, page 36 / 55 / ​​29

[0056] In a second stage 620 of the operation, the control unit 410 sends instructions to the hyperspectral camera 320 (the second type camera) in the second type of applicator / imager unit 160 to cause the hyperspectral camera to create a second image within the third image area 324 (Figure 4). As discussed above, the hyperspectral camera has a higher spectral resolution, as illustrated in Figure 5. The image information is communicated to the control unit via the control and data bus 430 or via the LAN (not shown), according to the communication configuration. The instructions sent to the hyperspectral camera are synchronized with the instructions sent to the RGB camera in the first stage 610, so that the two images include the same plant 114, if any, so that the two images can be compared as described above.

[0057] In a third step 630 of the operation, the control unit 410 correlates the first image from the RGB camera 310 in the second type of applicator / imager unit 160 with the second image from the hyperspectral camera 320 in the second type of applicator / imager unit to generate a correlation coefficient or correlatable features (mapping data) that map the low-resolution color information in the first image to the higher-resolution color information in the second image, as described above.

[0058] In a fourth stage 640 of the operation, the control unit 410 sends respective instructions to each of the RGB cameras 210 in the first type of applicator / imager units 150 to cause each RGB camera to create an image within the respective first image areas 214 (Figure 2). Each image may include an image of a second respective plant 114. The image information from each RGB camera in each of the first type of applicator / imager units is communicated to the control unit via the respective bus. Petition 870250072191, dated 08 / 15 / 2025, page 37 / 55 / ​​29 control and data 430 or via LAN (not shown) according to the communication configuration.

[0059] In a fifth step 650 of the operation, the control unit 410 applies the correlation coefficient or correlatable features (mapping data) generated in the third step 630 to the low-resolution color information in each of the first images generated in the fourth step 640 to produce a respective second generated hyperspectral image.

[0060] In a sixth step 660 of the operation, the control unit 410 performs spectral analysis on the second image of the first plant 114 in the third image area 324 (Figure 4) to determine a respective characteristic of the plant. For example, when the agricultural application system 100 (Figure 1) is configured to apply a herbicide to unwanted plants (e.g., weeds), the spectral analysis can identify the plant as a weed, and the control unit will activate the controllable spraying mechanism 200 in the second type of applicator / imaging unit 160 to spray the herbicide on the plant.In another example, when the agricultural application system is configured to selectively apply fertilizer, another nutrient, an insecticide, or similar substance to a plant, the result of spectral analysis can determine whether the plant's condition indicates that it needs the material to be sprayed and, if so, selectively activate the controllable spraying mechanism in the second type of applicator / imaging unit to apply the material to the plant.

[0061] In a seventh step 670 of the operation, the control unit 410 performs spectral analysis on the second hyperspectral image generated of each second plant 114 in each respective first image area 214 (Figure 2) to determine a respective plant characteristic. As described above, the applied material may be a herbicide for a weed. Petition 870250072191, dated 08 / 15 / 2025, page 38 / 55 / ​​29 weed or beneficial material for a cultivated plant.

[0062] The method illustrated in Figure 6 is repeated as the agricultural application system 100 advances through the field 110.

[0063] It should be understood from the preceding description that the operations performed by control unit 410 can be performed in a centralized control unit, as illustrated. Alternatively, the processing described above can be distributed among the devices described. For example, when communication between devices is implemented as a CAN bus, each first type of applicator / imager unit 150 and the second type of applicator / imager unit 160 can include an integrated controller (not shown) with a CAN bus-compatible interface (not shown). Thus, the control unit can send the correlation coefficient or correlatable features (mapping data) to each first type of applicator / imager unit so that the integrated controller can generate a respective hyperspectral image from the RGB data generated by the respective RGB camera.Alternatively, each onboard controller can perform spectral analysis on the respective hyperspectral image and control the respective controllable spraying mechanism 200. In this mode, the control unit initializes each onboard controller with the parameters necessary to identify the expected plants and characterize the plant conditions.

[0064] It should be understood from the preceding description that, by having the hyperspectral camera 320 and the associated RGB camera 310 in the second type of applicator / imaging unit 160 carried on the support structure 140 with the RGB cameras 210 in the first type of applicator / imaging unit 150, any changes in image conditions (e.g., time of day, cloud cover, type of crop being imaged, soil type, field residue or the like) will be incorporated into the generation of a Petition 870250072191, dated 08 / 15 / 2025, page 39 / 55 / ​​29 updated correlation coefficient or updated correlatable features. The updated correlation coefficient or updated correlatable feature is applied to the first images from the RGB cameras in the first type of applicator / imager units.

[0065] The previous use of a correlation coefficient or correlatable features to leverage relatively low spectral resolution cameras to create higher spectral resolution images can also be used to implement relatively inexpensive cameras for other measurement purposes. For example, the second type of applicator / imager unit 160 can include a stereo camera like the second type's 320 camera to provide convolutional neural network (CNN) depth measurements. A correlation coefficient or correlatable features can be generated by comparing the images and depth measurement with the images from the first type's 310 monocular RGB camera, so that the 210 RGB cameras in the first type of applicator / imager units 150 can also be used to provide respective depth measurements.

[0066] As another example of using a correlation coefficient or correlatable features to leverage low-resolution cameras, an ultrasonic height sensor can be included in the second type of applicator / imager unit 160. The height measurement on the Z-axis in combination with the area measurements on the X and Y axes can be used to determine the actual biomass of the plant photographed by the RGB camera 310 in the second type of applicator / imager unit. It can be assumed that plants photographed by the RGB cameras 210 in the first type of applicator / imager unit 150 with the same area measurements have the same height and corresponding biomass. Although described here as an agricultural application system with applicator devices, the preceding description also applies to a crop imaging system that assesses plant characteristics without applying material in response to the assessment results. By. Petition 870250072191, dated 08 / 15 / 2025, page 40 / 55 / ​​29 example, the system described here can generate and store geolocation data based on camera outputs and evaluations of camera outputs.

[0067] Thus, it is observed that the apparatus and methods of the present description readily achieve the aforementioned ends and advantages, as well as those inherent therein. Although certain preferred embodiments of the description have been illustrated and described for the purposes present, numerous changes in the arrangement and construction of parts and steps may be made by those skilled in the art, changes which are encompassed within the scope and spirit of the present description, as defined by the appended claims. Each described feature or embodiment may be combined with any of the other described features or embodiments. Petition 870250072191, dated 08 / 15 / 2025, page 41 / 55

Claims

1 / 6 CLAIMS 1. A method for using a plurality of low spectral resolution image-generating devices (210, 310) to determine features of a plurality of plants (114), characterized in that the method comprises: generating a respective first image of a first plant (114) using a first image-generating device (310) having a first spectral resolution; generating a second image of the first plant (114) using a second image-generating device (320) having a second spectral resolution, the second spectral resolution being higher than the first spectral resolution; correlating the second image of the first plant (114) with the first image of the first plant (114) to generate a correlation coefficient or correlatable features by mapping the first image of the first plant (114) to the second image of the first plant (114);generate a respective first image of at least one plant (114) in a group of plants (114) using a respective third image generation device (210) for each respective plant (114) in the group of plants (114), each third image generation device (210) having the first spectral resolution; apply the correlation coefficient or correlatable features to the respective first image of at least one plant (114) in the group of plants (114) to produce a respective second generated image of at least one plant (114) in the group of plants (114), the respective second generated image of at least one plant (114) in the group of plants (114) having the second spectral resolution; analyze the spectral features of the second image of the first plant (114) to determine a respective physical feature of Petition 870250072191, dated 15 / 08 / 2025, page. 42 / 55 2 / 6 first plant (114);and analyze the spectral characteristics of the respective second image generated of at least one plant (114) in the group of plants (114) to determine a respective physical characteristic of at least one plant (114) in the group of plants (114).

2. Method according to claim 1, characterized in that: the first image generating device (310) and the third image generating device (210) are RGB cameras with three spectral resolution channels comprising a red channel, a green channel and a blue channel; and the second image generating device (320) is a hyperspectral camera with more than three spectral resolution channels.

3. Method according to claim 2, characterized in that the second image generation device (320) has at least 40 spectral resolution channels.

4. Method according to claim 1, characterized in that it further comprises: selectively activating a first application mechanism (200) positioned close to the first image generation device (310) to apply a material to the first plant (114) when the respective physical feature of the first plant (114) has a first feature; and selectively activating a respective second application mechanism (200) positioned close to a respective third image generation device (210) to apply the material to at least one plant (114) in the group of plants (114) when the respective physical feature of at least one plant (114) in the group of plants (114) has a second feature. Petition 870250072191, dated 15 / 08 / 2025, p. 43 / 55 3 / 6 5. Method according to claim 4, characterized in that the second characteristic is the same as the first characteristic.

6. Method according to claim 5, characterized in that the first characteristic is a type of plant (114).

7. Method according to claim 6, characterized in that: the plant type (114) is a weed; and the material applied to the first plant (114) and the second plant (114) is a herbicide.

8. Method according to claim 4, characterized in that: the first plant (114) is a crop plant (114), the first characteristic is a relative health of the first plant (114), and the material applied to the first plant (114) is beneficial to the health of the plant (114); and at least one plant (114) in the group of plants (114) is a crop plant (114), the second characteristic is a relative health of at least one plant (114) in the group of plants (114), and the material applied to at least one plant (114) in the group of plants (114) is beneficial to the health of at least one plant (114) in the group of plants (114).

9. Crop evaluation system, characterized in that the system comprises: a first machine visualization unit (310) of a first type oriented to use a first image acquisition technique to obtain a first image of a first plant (114), the first image of the first plant (114) having a first spectral resolution; a first machine visualization unit (320) of a second type positioned close to the first machine visualization unit (310) of the first type and oriented to use a second image acquisition technique to obtain at least one second image of the first plant (114) having a second spectral resolution, the second spectral resolution being greater than the first spectral resolution;a second machine visualization unit (210) of the first type oriented to use the first image acquisition technique to obtain a first image of a second plant (114), the first image of the second plant (114) having the first spectral resolution; a processing system (400) configured to: receive the first image of the first plant (114) and the second image of the first plant (114); map elements of the first image of the first plant (114) to elements of the second image of the first plant (114) to generate a correlation coefficient or correlatable features between the first image of the first plant (114) at the first spectral resolution and the second image of the first plant (114) at the second spectral resolution; receive the first image of the second plant (114);Apply the correlation coefficient or correlatable features to the elements of the first image of the second plant (114) to produce a second generated image of the second plant (114); analyze the spectral features of the second image of the first plant (114) to determine at least one physical feature of the first plant (114); and analyze the spectral features of the second generated image of the second plant (114) to determine at least one physical feature of the second plant (114).

10. Crop assessment system according to claim 9, characterized in that additionally: the crop assessment system (400) comprises Petition 870250072191, dated 15 / 08 / 2025, page 45 / 55 5 / 6 additionally a source (130) of sprayable material, and at least one first spraying unit (200) and a second spraying unit (200), wherein: the first spraying unit (200) and the second spraying unit (200) are coupled to receive the sprayable material from the source (130) of the sprayable material; the first spraying unit (200) is positioned close to the first machine viewing unit (310) of the first type and is positioned close to the first machine viewing unit (320) of the second type; the second spraying unit (200) is positioned close to the second machine viewing unit (210) of the first type;the first spraying unit (200) is controllable to selectively spray the sprayable material in a first plant (114) close to the first spraying unit (200) in response to receiving a first command; and the second spraying unit (200) is controllable to selectively spray the sprayable material in a second plant (114) close to the second spraying unit (200) in response to receiving a second command; and the processing system (400) is further configured to: selectively send the first command to the first spraying unit (200) to activate the first spraying unit (200) in response to at least one determined physical characteristic of the first plant (114);and selectively send the second command to the second Petition 870250072191, dated 08 / 15 / 2025, page 46 / 55 6 / 6 spraying unit (200) to activate the second spraying unit in response to at least one determined physical characteristic of the second plant (114).; 11. Crop evaluation system according to claim 10, characterized in that at least one physical characteristic of the second plant (114) is a type of plant (114).

12. Crop evaluation system according to claim 11, characterized in that: the type of the second plant (114) is an undesirable plant (114); and the material applied to the second plant (114) is a herbicide.

13. Crop evaluation system according to claim 10, characterized in that: at least one physical characteristic of the second plant (114) is a relative health of the second plant (114); and the material applied to the second plant (114) is a material beneficial to the health of the second plant (114). Petition 870250072191, dated 15 / 08 / 2025, pp. 47 / 55

Images