DYNAMIC PERCEPTION SYSTEM CALIBRATION

Abstract

A system comprises: a perception system associated with a working machine and configured to capture an image displaying one or more properties; a perception position actuator controllable to change the position of the perception sensor during the operation of the working machine; one or more processors; and a memory that stores instructions executable by the one or more processors. When executed by the one or more processors, the instructions cause the one or more processors to: perform a calibration operation during the operation of the working machine and in response to a calibration trigger, in order to generate a calibration transformation; and control the working machine, at least on the basis of the calibration transformation and the image.

Description

AREA OF DESCRIPTION

[0001] This description concerns the activities of working machines. More specifically, it concerns perception sensor systems used in the control of working machines, such as agricultural machinery. BACKGROUND

[0002] There is a wide variety of different types of working machines. These include agricultural machines, such as, but not limited to, agricultural harvesting vehicles (e.g., combine harvesters, etc.). A working machine, such as an agricultural machine, can be controlled during an operation based on sensor data, for example, sensor data generated by sensors associated with the working machine. An example of such sensors are sensory sensors (e.g., cameras), which generate one or more images as sensor data that can be used to control the working machine.

[0003] The foregoing explanation is provided for general background information only and is not intended to be used as an aid in determining the scope of protection of the claimed subject matter. SUMMARY

[0004] A system comprises: a perception system associated with a working machine and configured to acquire an image displaying one or more properties; a perception position actuator controllable to change the position of the perception sensor during the operation of the working machine; one or more processors; and a memory that stores instructions executable by the one or more processors. When executed by the one or more processors, the instructions cause the one or more processors to: perform a calibration operation during the operation of the working machine and in response to a calibration trigger, in order to generate a calibration transformation; and control the working machine, at least on the basis of the calibration transformation and the image.

[0005] This summary is provided to present a selection of concepts in simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential characteristics of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of protection of the claimed subject matter. The claimed subject matter is not limited to implementations that resolve any or all of the disadvantages specified in the Background section. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a partly pictorial, partly schematic representation of an exemplary working machine in the form of an agricultural harvesting vehicle. Fig. 2 is a pictorial representation that provides an example of the in Fig. The image shows one agricultural harvesting vehicle. Fig. Figure 3 is a pictorial representation that provides an example of what is in Fig. The image shows one agricultural harvesting vehicle. Fig. Figure 4 is a block diagram of an example of a system architecture. Fig. Figure 5 is a block diagram illustrating some examples of system architecture components, including a dynamic perception monitoring system. Fig. 6A and Fig. 6B (summarized here as Fig. Figure 6) shows a flowchart illustrating an exemplary operation of the system architecture during the performance of dynamic perception monitoring, calibration and machine control. Fig. Figure 7 is a block diagram showing an example of elements of a system architecture in communication with a remote server architecture. Fig. 8, Fig. 9 and Fig. Figure 10 shows examples of mobile devices that can be used in a system architecture. Fig. Figure 11 is a block diagram showing an example of a computing environment that can be used in a system architecture. DETAILED DESCRIPTION

[0006] For a better understanding of the principles of this disclosure, reference is now made to the examples illustrated in the drawings, which are described in specific language. Nevertheless, it is understood that this is not intended to limit the scope of protection of the disclosure. Any modifications and further alterations to the described devices, systems, methods, and any further application of the principles of this disclosure that are normally apparent to a person skilled in the art in the field to which the disclosure relates are all included. In particular, it is fully taken into consideration that the features, components, and / or steps described with reference to one example may be combined with the features, components, and / or steps described with reference to other examples of this disclosure.

[0007] As discussed above, the operating parameters of a machine can be controlled during operation based on sensor data generated by sensors associated with the machine (e.g., attached to it or otherwise communicating with it). A control system can use this sensor data to control one or more of the machine's operating parameters.

[0008] Examples of such sensors include perception sensors, such as cameras, LiDAR sensors, radar sensors, and other types of perception sensors. Cameras can be one or more different types. Perception sensors generate one or more images as sensor data, for example, a series of images, which can be used to control the machine.

[0009] In some current systems, perception sensors are permanently mounted or integrated into the machine at fixed locations, which helps to reduce variability and system errors. For example, perception sensors generally require calibration to ensure their accuracy. If the perception sensors are in a fixed position, calibration can be performed throughout the entire duration of operation. However, there may be a number of reasons for wanting to move a perception sensor, such as to change the field of view or to reduce occlusions. Fixed perception sensors have limited ability to change the field of view or reduce occlusions, at least compared to movable perception systems.To enable multiple fields of view and reduce occlusions, current systems can use a larger number of perception sensors distributed across the machine, which can increase costs.

[0010] Perception sensors capable of dynamically moving during operation would offer greater freedom in changing the field of view and reducing occlusions with fewer sensor systems. However, when a perception sensor is moved from one position (e.g., location, orientation, etc.) to another, calibration of the sensor may be necessary to ensure system accuracy (e.g., accurate processing and extraction of values ​​from the generated sensor data, such as images).

[0011] This document describes systems and methods that enable the dynamic movement and calibration (or recalibration) of sensory sensors during a process. The dynamically moved and recalibrated sensory sensors can generate sensor data (e.g., images) that can be used to control an associated machine.

[0012] It is understood that, although the examples given here relate to agricultural machinery such as agricultural harvesting vehicles, the systems and methods disclosed herein are applicable to and can be used with a variety of other types of machinery, including a variety of other types of agricultural machinery.

[0013] Fig. Figure 1 is a partly pictorial, partly schematic representation of an exemplary agricultural machine 100 in the form of an agricultural harvesting vehicle 100-1. The example of Fig. Figure 1 shows the agricultural harvesting vehicle 100-1 in the form of a combine harvester. As in Fig. As shown in Figure 1, the harvesting vehicle 100-1 comprises ground-contacting traction elements (wheels or tracks) 144 and 145, which may be driven by a drive system (e.g., internal combustion engine, electric motors, hydrostatic drive, and other drivetrain elements such as a gearbox) to propel the harvesting vehicle 100 across a work site 10 (e.g., a field). The harvesting vehicle 100-1 includes an operator compartment or cab 119, which incorporates a variety of different operator interface mechanisms (e.g., 218 shown in Figure 1). Fig. 4) for controlling the harvesting vehicle 100-1 and for presenting (e.g., displays, etc.) various information. The harvesting vehicle 100-1 comprises an inclined conveyor 106, a feed accelerator 108, and a threshing unit generally designated 110. The inclined conveyor 106 and the feed accelerator 108 are part of a material handling subsystem 125. The harvesting header 104 is pivotally coupled to a frame 103 of the harvesting vehicle 100-1 along a pivot axis 105. One or more actuators 107 drive the movement of the harvesting header 104 about the axis 105 in the direction generally indicated by the arrow 109. Thus, the vertical position of the harvesting header 104 (the harvesting header height) above a floor 111, over which the harvesting header 104 moves, can be controlled by actuating the actuator 107. Although in Fig. Not shown in Figure 1, the agricultural harvesting vehicle 100-1 may also include one or more actuators that apply an angle of inclination, a tilt angle or both to the harvesting header 104 or sections of the harvesting header 104.

[0014] The material handling subsystem 125 further comprises a threshing unit 110 with a threshing rotor 112 and a series of threshing concaves 114. The material handling subsystem 125 also includes a separator 116. The agricultural harvesting vehicle 100-1 further comprises a cleaning subsystem or sieve box (collectively referred to as the cleaning subsystem 118), which includes one or more cleaning blowers 120, a chaff sieve 122, and a fine sieve 124. The material handling subsystem 125 further comprises an unloading drum 126, a return elevator 128, and a clean grain elevator 130. The clean grain elevator conveys clean grain into a material hopper (or clean grain tank) 132.

[0015] The combine harvester 100-1 also includes a material transfer system comprising a conveying mechanism 134 and an unloading auger 135. The unloading auger 135 includes a discharge spout 136. In some examples, the discharge spout 136 can be movably coupled to the unloading auger 135, allowing the discharge spout 136 to be rotated in a controlled manner to change its orientation. The conveying mechanism 134 can be of various different types, such as an auger, a blower, or a belt conveyor. The conveying mechanism 134 is connected to the clean grain tank 132 and is driven (e.g., by an actuator such as an electric motor or an internal combustion engine) to convey material from the grain tank 132 through the unloading auger 135 and the discharge spout 136. The unloading pipe 135 can be extended from a stowed position via a series of positions (in Fig. (1 shown) away from the agricultural harvesting vehicle 100-1, the discharge spout 136 can be rotated into a variety of fold-out positions to align it with a material intake of a material receiving machine designed to receive the material in the grain tank 132. In some examples, the discharge spout 136 is also rotatable by an actuator to adjust the direction of the material flow exiting the discharge spout 136.

[0016] The harvesting vehicle 100-1 also includes a residue separation system 138, which can include a chopper 140 and a spreader 142.

[0017] In some examples, a harvesting vehicle within the scope of this disclosure may have more than one of the aforementioned subsystems. In some examples, the harvesting vehicle 100-1 may have a left and a right cleaning subsystem, a left and a right separator, etc., which are arranged in Fig. 1 are not shown.

[0018] During operation, the harvesting vehicle 100-1 moves, for example, across a work site (e.g., a field) 10 in the direction indicated by arrow 147. While the harvesting vehicle 100-1 is moving, the harvesting header 104 engages the crop plants to be harvested and cuts them with a cutter bar 107 on the harvesting header 104 to produce cut crop material.

[0019] The cut crop material is captured by a transverse conveyor (e.g., auger, conveyor belts, etc.) 113, which conveys the separated crop material to the center of the header 104. There, the separated crop material is transported through an opening to a conveying device in the inclined conveyor 106, leading to the feed accelerator 108, which accelerates the separated crop material into the threshing unit 110. The separated crop material is threshed by the rotor 112, which rotates the crop material against threshing concaves 114. The threshed crop material is moved by a separator rotor in the separator 116, with a portion of the residue being moved by the discharge drum 126 to the residue separator 138. The portion of the residue that is transferred to the residue sorting system 138 is chopped by the residue chopper 140 and distributed on the field by the spreader 142.In other configurations, the residue is spread in a swath by the agricultural harvesting vehicle 100-1.

[0020] Grain falls into the cleaning subsystem 118. The chaff screen 122 separates some larger pieces of NKB (non-ferrous granules) from the grain, and the screen 124 separates some finer pieces of NKB from the grain. The grain then falls onto a conveyor (e.g., screw conveyor, etc.) that transports the grain to an inlet end of the grain elevator 130, and the grain elevator 130 conveys the grain upwards and deposits it in the grain tank 132. Residue is removed from the cleaning subsystem 118 by an airflow generated by one or more cleaning blowers 120. The cleaning blowers 120 direct air along an airflow path upwards through the fine screens and the chaff screens. The airflow transports the residue in the harvester 100-1 to the rear of the residue handling subsystem 138.

[0021] The return elevator 128 directs the returned material back to the threshing unit 110, where it is threshed again. Alternatively, the returned material can also be conveyed by a return elevator or other transport device to a separate re-threshing mechanism, where it is also threshed again.

[0022] The 100-1 harvesting vehicle can include a variety of sensors, some of which are integrated into Fig. 1 are shown.

[0023] The ground speed sensor 146 detects the ground speed of the harvesting vehicle 100-1. The ground speed sensor 146 can detect the ground speed of the harvesting vehicle 100-1 by measuring the rotational speed of the ground-contacting traction elements 144 and / or 145, a drive shaft, an axle, or other components. In some cases, the ground speed can be measured using a positioning system, such as a global positioning system (GPS), a dead reckoning system, a long-range navigation system (LORAN system), a Doppler velocity sensor, or a wide variety of other systems or sensors that provide a ground speed reading.The ground speed sensors 146 can also include direction sensors such as a compass, magnetometer, gravimetric sensor, gyroscope, or GPS to determine the direction of travel in two or three dimensions in combination with the speed. If the harvester 100-1 is on a slope, its orientation relative to the slope is thus known. An orientation of the harvester 100-1 could be, for example, uphill, downhill, or perpendicular to the slope.

[0024] The harvesting vehicle 100-1 also includes one or more perception sensor systems 148. Each perception sensor system 148 includes one or more perception sensors 150 (e.g., cameras (e.g., mono cameras, stereo cameras, color cameras (e.g., RGB cameras), multispectral cameras, thermal cameras, infrared cameras, etc.), LiDAR sensors, radar sensors, etc.). The perception sensors 150 can, for example, capture one or more images of the operating location 10, the machine 100, or the environment of the operating location 10 and the machine 100 in order to detect one or more corresponding properties.

[0025] Each perception sensor system 148 further comprises an actuable support structure 152 (e.g., an actuable mounting structure) and one or more actuators 154 (e.g., a linear actuator, a motor, a fluid actuator, etc.) that drive a movement of the actuable support structure 152 and thus a movement of one or more corresponding perception sensors 150. The actuators 154 can be used to change the position (height, position, or orientation, or a combination thereof) of one or more corresponding perception sensors 150. It is understood that the type of actuator 154 and the type of support structure 152 can vary. As described in Fig. As shown in Figure 1, the support structure 152 includes, for example, a movable arm (e.g., a telescopic arm, a robot arm, etc.), but in other examples it can also take on other forms or include additional elements. For example, but not exclusively, a tiltable and / or rotatable element that allows a change in the orientation (e.g., pitch, roll, yaw angle) of the perception sensors 150.

[0026] Even if Fig. Figure 1 shows some exemplary mounting locations for perception sensors 150; it is understood that the perception sensors 150 can alternatively or additionally be mounted at a variety of other locations on a working machine 100, e.g. the harvesting vehicle 100-1.

[0027] A work machine 100, such as the harvester 100-1, as well as other work machines, can include various other sensors, some of which are in Fig. 4 will be described. A working machine 100, such as the harvesting vehicle 100-1, as well as other working machines, can include various other elements, some of which are described in Fig. 4 will be described.

[0028] Fig. 2A and Fig. Figure 2B shows an example of a change in the position of a perception sensor 150. As in Fig. As shown in Figure 2A, the perception sensor 150 is in a first position with a first field of view represented by lines 180. In the example shown, the perception sensor 150 generates images of the crop 190 in order to recognize properties of the crop 190. Fig. In step 2B, the actuators 154 were activated to move the perception sensor 150 from the first position to a second position (different height than the first position) with a second field of view, represented by lines 182. In the Fig. 2A and Fig. In the examples shown in 2B, the perception sensor 150 may be moved from the first position to the second position to take into account an obstruction to the view, or to take better pictures of the ears of the crop plants 190, or to focus on another property of the crop plants 190, or for a variety of other reasons.

[0029] Fig. 3A and Fig. Figure 3B are pictorial representations showing an example of a change in the position of a perception sensor 150. As in Fig. As shown in Figure 3A, the working machine 100 (e.g., the harvesting vehicle 100-1) comprises a plurality of perception sensor systems 148 (for example, 148-1 and 148-2). The perception sensor system 148-1 comprises a perception sensor 150-1 (e.g., a stereo camera), and the perception sensor system 148-2 comprises a perception sensor 150-2 (e.g., a stereo camera). As shown in Figure 3A, the working machine 100 (e.g., the harvesting vehicle 100-1) comprises a plurality of perception sensor systems 148 (e.g., 148-1 and 148-2). The perception sensor system 148-1 comprises a perception sensor 150-1 (e.g., a stereo camera), and the perception sensor system 148-2 comprises a perception sensor 150-2 (e.g., a stereo camera). Fig. As shown in Figure 3A, the perception sensor 150-1 is in a first position with a first field of view represented by lines 184. Fig. In step 3B, the actuators 154 were activated to move the perception sensor 150-1 from the first position to a second position (different orientation (e.g., yaw angle) than the first position) with a second field of view, which is represented by lines 186. In the Fig. 3A and Fig. In the examples shown in 3B, the perception sensor 150 may be moved from the first position to the second position to take into account a visual obstruction, or to produce better (e.g. higher quality) images of a feature of interest, or to focus on another feature of interest, or for a variety of other reasons.

[0030] Fig. Figure 4 is a block diagram showing an exemplary system architecture 500 (hereinafter also referred to as System 500). If the work machine 100 is an agricultural work machine (e.g., the harvester 100-1, etc.), System 500 can also be referred to as the agricultural system. System 500 comprises one or more work machines 100. System 500 also comprises one or more remote computing systems 300, one or more networks 359, one or more interface mechanisms for remote users 364, and may also include a variety of other elements 202.

[0031] Each working machine 100 itself comprises, for example, one or more processors or servers 201, one or more data storage devices 204, a communication system 206, one or more actuated perception sensor support structures 207, one or more sensors 208, a control system 214, one or more controllable subsystems 216, one or more operator interface mechanisms 218, and various other elements and functionalities 219.

[0032] The remote computing systems 300 comprise, as illustrated, one or more processors or servers 301, one or more data storage devices 304, a communication system 306, and may include various other elements and functionalities 319.

[0033] Data stores 204 and 304 each store a variety of data (generally referred to as data 205 and data 305, respectively), some of which are described in more detail below. For example, data 205, data 305, or data 405, or a combination thereof, can include sensor data, calibration data, machine data, site data, and various other data, including, but not limited to, several other data described herein. Some examples of the different data are given in Fig. 5 described in more detail. Additionally, the data 205 may include computer-executable instructions that can be executed by one or more processors or servers 201 to implement other elements or functionalities of the system 500, including other elements or functionalities of the work machines 100. Additionally, the data 305 may include computer-executable instructions that can be executed by one or more processors or servers 302 to implement other elements or functionalities of the system 500, including other elements of the remote computing systems 300. It is understood that the data storage 204 and data storage 304 may include various forms of data storage, for example, both volatile data storage (e.g., random access memory (RAM)) and non-volatile data storage (e.g., read-only memory (ROM), hard disks, solid-state drives, etc.).

[0034] The actuated perception sensor support structures 207 comprise actuated (e.g., movable) structures for attaching perception sensors 226 to a machine 100 and for dynamically changing the position of the perception sensors 226. The actuated perception sensor support structures 207 can be actuated by actuating perception sensor position actuators 254. The actuated perception sensor support structures 207 can include one or more movable arms (e.g., telescopic arms, robot arms, etc.), tiltable and / or rotatable elements (e.g., rotatable and alignable mounting platforms (e.g., tables, etc.)), and other elements. In other examples, the actuated perception sensor support structures 207 can take on various other forms. The actuated perception sensor support structures 207 enable a change in the position of the perception sensors 226 in one or more degrees of movement.For example, the actuated perception sensor support structures 207 can enable movement of a perception sensor in three dimensions of a three-dimensional coordinate system (e.g., X, Y, and Z axes), for example, along a Z-axis corresponding to a change in the height of the perception sensor 226, along an X-axis corresponding to a change in the horizontal position of the perception sensor 226 in a side-to-side plane (e.g., distance to the left or right side of the machine 100 in a side-to-side direction), and along a Y-axis corresponding to a change in the horizontal position of the perception sensor 226 in a forward-backward plane (e.g., further away from or closer to the machine 100 in a forward-backward direction). In another example, the actuated perception sensor support structures 207 can enable a change in the orientation of the perception sensors 226, such as...a change in one or more roll, pitch, or yaw angles. An example of actuated perception sensor support structures 207 are those in . Fig. Supporting structures shown in 1 to 3B 152.

[0035] The sensors 208 can include one or more perception sensors 226, one or more heading / velocity sensors 225, one or more geoposition sensors 203, one or more perception sensor motion sensors 227, one or more perception sensor motion criteria sensors 228, and also various other sensors 229. The perception sensor motion criteria sensors 228 can include one or more environmental sensors 240, one or more machine condition sensors 242, one or more power sensors 244, one or more user input sensors 246, and also various other sensors 247. The sensor data generated by the sensors 208 can be transmitted to remote computing systems 300 and to other working machines 100.

[0036] The control system 214 itself can include one or more controllers 235 for controlling various other elements of a working machine 100, the dynamic perception monitoring system 215, and also other elements 237. The controllable subsystems 216 can include the drive subsystem 250, the steering subsystem 252, the perception sensor position actuators 254, the actuators 256, and various other subsystems 258.

[0037] The heading / speed sensors 225 detect a heading property (e.g., direction of travel) and / or speed property (e.g., travel speed, acceleration, deceleration, etc.) of a working machine 100. This can include sensors that detect the movement (e.g., rotational movement) of ground-contacting traction elements (e.g., wheels or tracks) or the movement of components coupled to the ground-contacting elements or other elements (e.g., axles), or that can utilize received signals from other sources, such as geoposition sensors. Thus, while the heading / speed sensors 225 described here are shown separately from geoposition sensors 203, in some examples the heading / speed of the machine is derived from signals coming from geoposition sensors 203 and subsequent processing.In other examples, the 225 course / speed sensors are separate sensors and do not use signals received from other sources. An example of 225 course / speed sensors are those in [reference to relevant document]. Fig. 1 of the sensors shown 146.

[0038] The geoposition sensors 203 detect or recognize, for example, the geoposition or geographic location of a work machine 100. The geoposition sensors 203 may include, among other things, a receiver of a global navigation satellite system (GNSS) that receives signals from a GNSS satellite transmitter. The geoposition sensors 203 may also include a real-time kinematic (RTK) component configured to improve the accuracy of position data derived from the GNSS signal. The geoposition sensors 203 may include a dead reckoning navigation system, a cellular triangulation system, or any of a variety of other geoposition sensors.

[0039] The perception sensors 226 comprise image acquisition (or generation) mechanisms such as cameras (e.g., monocular cameras, stereo cameras, color cameras (e.g., RGB cameras), multispectral cameras, thermal cameras, infrared cameras, etc.), LiDAR sensors, radar sensors, etc. The perception sensors 226 capture (or generate) images of the work machine 100, the operating location (e.g., 10) where the work machine 100 operates, or the environment of the work machine 100 or the operating location in order to capture its properties. The images captured (or generated) by the perception sensors 226 display the properties (and their values) and can be used for controlling the work machine 100. An example of perception sensors 226 are those described in Fig. 1 to 3B shown perception sensors 150.

[0040] The perception sensor motion sensors 227 generate sensor data indicating the movements of one or more perception sensors 226. The perception sensor motion sensors 227 can be software-based sensors that detect a perception sensor motion command (e.g., a control signal) issued by the control system 214 (e.g., by the controllers 235), such as a perception sensor motion command that controls the perception sensor position actuators 254. The perception sensor motion sensors 227 can also be hardware sensors that detect movements of the perception sensors 226 or of elements associated with the perception sensors 226 (such as actuators 254 or support structures 207), such as rotary encoders, transducers, potentiometers, and a variety of other types of hardware sensors.

[0041] The perception sensor motion criteria sensors 228 detect various criteria for controlling movements (changes in position) of the perception sensors 226. The environment sensors 240 detect environment-related perception sensor motion criteria, such as weather characteristics that can affect the performance (e.g., detection or image quality, etc.) of the perception sensors 226, such as wind characteristics (e.g., wind speed and direction), ambient light characteristics (e.g., intensity or amount of ambient light, angle of inclination of ambient light, etc.), precipitation characteristics (e.g., amount and type of precipitation), temperature, and other weather characteristics. It becomes apparent that in some examples, at least some weather characteristics can be obtained from sources other than the weather sensors, such as publicly accessible external weather sources (e.g., internet-based sources) via a communication system (e.g.,206 or 306) via networks 359. The environmental sensors 240 can detect environment-related perception sensor motion criteria, such as objects or other items that can act as visual obstructions for the perception sensors 226, such as dust and debris, trees, power lines, components of the work machine 100, and various other objects at or near the work site. The environmental sensors 240 can detect various other environment-related perception sensor motion criteria.

[0042] The machine state sensors 242 detect machine state-related perception sensor movement criteria that can be used to determine whether a perception sensor 226 should be moved, such as the detection of auxiliary operations of the working machine 100 (e.g., the unloading operation performed by the material transfer subsystem of the harvesting vehicle 100-1). For example, it may be desirable or undesirable to move the perception sensors 150 while a subsystem of a working machine 100 is performing an auxiliary operation. The machine state sensors 242 can detect machine state-related perception sensor movement criteria that can be used to determine whether a perception sensor 150 should be moved, such as the detection of positions of components of the working machine 100 (e.g., the position or height of the harvesting header 104 of the harvesting vehicle 100-1).For example, depending on the position of a component of the working machine 100, it may be desirable or undesirable to move the perception sensors 226; for example, it may be undesirable to move a perception sensor 226 if this could obstruct the view of the perception sensor 226 or cause a collision between the perception sensor 226 and the component of the working machine 100, given the position of the component of the working machine 100.

[0043] The power sensors 244 detect machine-performance-related perception sensor movement criteria that can be used to determine whether a perception sensor 226 should be moved, such as one or more performance characteristics of the working machine 100 (e.g., the harvester 100-1). With regard to a working machine 100 in the form of a harvester 100-1, the one or more performance characteristics could include, for example, yield, crop loss (e.g., grain), efficiency (e.g., power consumption, fuel consumption, time to completion, etc.), crop cleanliness (e.g., grain), and a variety of other performance characteristics. It is understood that the performance characteristics detected by the power sensors 244 can vary depending on the type of working machine 100.Furthermore, the performance characteristics detected by the power sensors 244 can be used to determine whether a perception sensor 226 should be moved, particularly if the perception sensor 226 provides sensor data used in controlling the machine (i.e., generates sensor data used in control that affects performance). For example, if one or more performance characteristics are satisfactory (e.g., with respect to a threshold or a desired level), moving a perception sensor 226 may be undesirable, whereas in cases where one or more performance characteristics are unsatisfactory (e.g., with respect to a threshold or a desired level), moving a perception sensor 226 may be desirable. If performance is unsatisfactory, moving a perception sensor 226 may provide higher-quality sensor data (e.g.,less obstructed view, higher resolution, etc.) or provide sensor data that is more useful for improving performance (e.g., sensor data that points to or focuses more on a particular feature of interest).

[0044] The user input sensors 246 detect user input-related perception sensor movement criteria that can be used to determine whether a perception sensor 226 should be moved, such as user input (or user interactions) with an interface mechanism (e.g., 218, etc.) that controls movements of a perception sensor 226. The user input sensors 246 can detect user input (or user interactions) with an interface mechanism (e.g., 218, etc.) that controls movements of a perception sensor 226. The user input sensors 246 can be software-based sensors that detect user input (or user interactions) with an interface (e.g., a touchscreen or microphone) that commands movements of a perception sensor 226.Such user inputs (or user interactions) can be an interaction with a button on the touchscreen that commands a movement or provides one or more values, or an interaction with a microphone that acoustically commands a movement or provides one or more values. The user input sensors 246 can be hardware sensors that detect user inputs (or user interactions) that cause a movement, change in position, or rotation of an interface mechanism (e.g., 218, etc.), such as a button, knob, lever, or other movable interface mechanism that commands movements of a perception sensor 226. Such user inputs (or user interactions) can be interactions with movable interface mechanisms (e.g., button, knob, lever, etc.) that command a movement or provide one or more values.

[0045] The perception sensor motion criteria sensors 228 can include various other sensors 247 that detect various other perception sensor motion criteria.

[0046] The sensors 208 can also include various other types of sensors 229.

[0047] The control system 214 can include one or more controllers 235 (e.g., electronic control units that include or can be implemented by one or more processors, such as one or more processors 201) that generate control signals for controlling one or more components of a working machine 100 and / or components of the system 500. For example, but not exclusively, the controllers 235 can be a communication system controller for controlling the communication system 206, an interface controller for controlling one or more interface mechanisms (e.g.,218 or 364 or both), a drive controller for controlling the drive subsystem 250 to control a travel speed of a working machine 100, a path planning controller for controlling the steering subsystem 252 to control a route or course of a working machine 100, one or more perception sensor position actuator controllers for controlling the perception sensor position actuators 254, and one or more actuator controllers for controlling an operation of the actuators 454. In other examples, a central controller 235 may be used to generate control signals to control a variety of controllable subsystems 216, and in some examples, other elements of the system 500 as well.

[0048] The drive subsystem 250 comprises one or more controllable actuators (e.g. internal combustion engine, motors, pumps, gearboxes, etc.) that drive the ground-contacting traction elements (e.g. wheels or tracks) of a working machine 100 in order to vary the travel speed of a working machine 100.

[0049] The steering subsystem 452 comprises one or more controllable actuators (e.g. electric actuators, hydraulic actuators, etc.) which can be controlled to operate the steering and thus the course of a working machine 100.

[0050] The perception sensor position actuators 254 can be a variety of different types of actuating elements that can be actuated to change a position (e.g., height, position, orientation, etc.) of the perception sensors 226, for example, by causing movements of the actuable perception sensor support structures 207. The actuators 254 can include electromechanical actuators (e.g., linear actuators) or fluid actuators (e.g., pneumatic, hydraulic, etc.), as well as various other actuators (e.g., motors, etc.). An example of the actuators 254 are the actuators 154.

[0051] The actuators 256 encompass a wide variety of actuator types that control operating parameters (e.g., position (e.g., height, position, orientation), speed, distance, etc.) of one or more components of a machine 100. The actuators 256 can include actuators that control the position (e.g., height, depth, or distance from another component of the machine or from the operating point) or orientation (e.g., pitch, roll, yaw angle, etc.) of components of a machine 100, as well as actuators that control the speed of movement (e.g., rotational speed, lifting speed, etc.) of components of a machine 100. Actuators 256 can include, among others, electric motors, valves, pumps, hydraulic actuators (e.g., hydraulic cylinders, etc.), pneumatic actuators (e.g., pneumatic cylinders, etc.), electrical actuators (e.g., linear actuators, etc.), and various other types of actuators.Insofar as the working machine 100 is an agricultural harvesting vehicle 100-1, the actuators can comprise 254 actuators which can be controlled in such a way that they control operating parameters of one or more in . Fig. Control 1 of the described components.

[0052] Fig. Figure 4 also shows that the control system 214 can include a dynamic perception monitoring system 215. The dynamic perception monitoring system 215 is capable of identifying one or more properties based on sensor data, controlling the operation (e.g., dynamic movements) of the perception sensors 226, and performing in-situ calibration (calibration during operation of the machine 100) of the perception sensors 226. The dynamic perception monitoring system 215 is capable of causing a change in the position of one or more perception sensors 226 during the operation of the machine 100 and dynamically calibrating (e.g., recalibrating) the one or more perception sensors 226 during the operation of the machine 100 in response to a change in their position.It is understood that "during operation of the machine 100" in this context means that the machine 100 is at least at a place of operation (e.g., field, etc.) and is switched on, and in some examples further, that the machine 100 is in motion, and in some examples, is in motion and performing a corresponding operation (e.g., harvesting vehicle 100-1). Thus, the systems and methods disclosed herein can dynamically calibrate (e.g., recalibrate) one or more perception sensors 226 during operation, which may include the dynamic calibration of one or more perception sensors 226 while the machine 100 is moving and, in some examples, also while performing a corresponding operation (e.g., harvesting vehicle 100-1). In this way, the perception sensors 226 can be dynamically moved and calibrated (recalibrated) during the operation of a machine 100.The dynamic perception monitoring system 215 is used in . Fig. 5 explained in more detail.

[0053] The communication system 206 is used to provide communication between components of a work machine 100 or with other elements of the system 500, such as remote computing systems 300, or other work machines 100 or user interface mechanisms 364, or a combination thereof. The communication system 306 is used for communication between the components of a remote computing system 300 or with other elements of the system 500, such as work machines 100, other remote computing systems 300, or user interface mechanisms 364, or a combination thereof.

[0054] Communication systems 206 and 306 can each include wired and / or wireless communication circuits, as well as wired and wireless communication components. In some examples, communication systems 206 and 306 can each be a system for communication over the Internet, a cellular communication system, a system for communication over a wide area network or local area network, a system for communication over a controller area network (CAN), such as a CAN bus, a system for communication over a CAN-FD (controller area network flexible data-rate), such as a CAN-FD bus, a system for communication over a near-field communication network, a system for communication over Ethernet, or a communication system designed for communication over a variety of other networks.Communication Systems 206 and 306 can also include a system that enables the downloading or transfer of information to and from an SD (Secure Digital) card or a USB (Universal Serial Bus) card, or both. Both Communication Systems 206 and 306 can utilize Network 359. Network 359 can be any network from a variety of different types, such as the Internet, a cellular network, a wide area network (WAN), a local area network (LAN), a CAN (controller area network), a CAN-FD (controller area network flexible data-rate), a near-field communication network, Ethernet, or one from a wide variety of other networks or communication systems.

[0055] Fig. Figure 4 shows that one or more operators 361 can operate the work machines 100. The operators 361 interact with operator interface mechanisms 218. The operator interface mechanisms 218 may, in some examples, include, among a wide variety of other types of control devices, joysticks, levers, a steering wheel, couplings, pedals, buttons, wireless devices (e.g., mobile computing devices, etc.), rotary controls, keypads, a display device (including a display screen), user-operated elements (such as icons, buttons, etc.) on a display device, a microphone, and a loudspeaker (where speech recognition and speech synthesis are provided). Where a touch-sensitive display system is provided, the operators 361 can interact with the operator interface mechanisms 218 via touch gestures. In addition, at least some of the operator interface mechanisms 218 may be used for presentation (e.g.,Display, acoustic representation, haptic representation, etc.) of various information may be used. The examples described above are provided as illustrative examples and are not intended to limit the scope of protection of this disclosure. Accordingly, other types of user interface mechanisms 218 may be used and are within the scope of protection of this disclosure.

[0056] Fig. Figure 4 also shows remote users 366 who communicate with work machines 100 and remote computing systems 300 via networks 359 using user interface mechanisms 364. The user interface mechanisms 364 may, in some examples, include, among a wide variety of other types of control devices, joysticks, levers, a steering wheel, couplings, pedals, buttons, wireless devices (e.g., mobile computing devices, etc.), rotary knobs, keypads, a display device (including a display screen), user-operable elements (such as icons, buttons, etc.) on a display device, a microphone, and a loudspeaker (where speech recognition and speech synthesis are provided). Where a touch-sensitive display system is provided, users 366 can interact with the user interface mechanisms 364 using touch gestures. Furthermore, at least some of the user interface mechanisms 364 may be used for presentation (e.g.,B. Display, acoustic representation, haptic representation, etc.) of various information. The examples described above are provided as illustrative examples and are not intended to limit the scope of protection of this disclosure. Accordingly, other types of user interface mechanisms 364 may be used and are within the scope of protection of this disclosure.

[0057] The remote computing systems 300 can be a wide variety of different types of systems or combinations thereof. For example, the remote computing systems 300 may be located in a remote server environment. Furthermore, the remote computing systems 300 may be remote computing systems such as mobile devices, a remote network, an agricultural management system, a supplier system, or a wide variety of other remote systems. In one example, farm machinery 100 may be remotely controlled by remote computing systems 300 and / or remote users 366. In some examples, operators 361 are on board (e.g., in an operator compartment such as a cabin) the farm machinery 100. In some examples, the operators 361 are remote from the farm machinery 100 and control the farm machinery 100 via one or more interface mechanisms (e.g.,one or more of 218), which are remote from the machines but operationally coupled to the machines (e.g. communicatively coupled, such as via networks 359).

[0058] It is understood that in some examples the elements in System 500 may be distributed in different ways, including those determined by the one in Fig. The example shown in section 4 may differ. For example, the dynamic perception monitoring system 215, which is located in Fig. The dynamic perception monitoring system 215, which is arranged on the work machines 100, can also be located at another location, e.g., on one or more remote computer systems 300, without being restricted to this. In further examples, the dynamic perception monitoring system 215 can be distributed across a work machine 100 and a remote computer system 300.

[0059] Fig. Figure 5 is a block diagram that illustrates examples of some of the components of System 500 as well as the information flow between the components in more detail.

[0060] As in Fig. As shown in Figure 5, data storage 204 or data storage 304, or a combination thereof, can contain as data (205 or 305) sensor data 501, calibration data 502, machine data 503, deployment location data 504, and various other data 510, including, but not limited to, other data described herein. In some examples, the location of the data may depend on the location of the dynamic perception monitoring system 215 (also referred to as system 215).

[0061] As in Fig. As shown in Figure 5, the monitoring system 235 comprises one or more data processing systems 330, a perception sensor motion command system 332, a calibration trigger identification system 334, a calibration system 336, a power identification system 340, a presentation generator 342, and various other elements and functions 359. As will be described in more detail below, the system 215 can be operated to produce one or more monitoring outputs 360.

[0062] The sensor data 501 comprises sensor data (e.g., images, sensor signals, etc.) generated by the sensors 208. The sensor data 501 may include perception sensor data generated by the perception sensors 226, perception sensor-motion sensor data generated by the perception sensor-motion sensors 227, geoposition sensor data generated by the geoposition sensors 203, heading / speed sensor data generated by the heading / speed sensors 225, perception sensor-motion criteria sensor data generated by the perception sensor-motion criteria sensors 228, and various other sensor data generated by other sensors 229.The perception sensor motion criteria sensor data may include environmental sensor data generated by the environmental sensors 240, machine condition sensor data generated by the machine condition sensors 242, machine power sensor data generated by the power sensors 244, user input sensor data generated by the user input sensors 246, and various other sensor data generated by other sensors 247.

[0063] The calibration data 502 can include calibration transformations (e.g., model, matrix, calibration parameters (e.g., sensor parameter values ​​such as intrinsic and extrinsic parameter values, etc.)) that are used to calibrate the perception sensors 226 or to calibrate sensor data (e.g., images) generated by the perception sensors 226. For example, a perception sensor 226 can have a home or standard position for which there is a stored basic calibration transformation and stored calibration parameters (e.g., sensor parameter values ​​such as intrinsic and extrinsic parameter values) in the calibration data 502. Thus, when the perception sensor 226 is in the home or standard position, the perception sensor 226 (or the sensor data it generates (e.g., images)) can be calibrated by the stored basic calibration transformation.If a perception sensor 226 is caused to change its position during the process, a new calibration transformation 338 is generated and stored as calibration data 502, which can be used to calibrate (recalibrate) the perception sensor 226 (or the sensor data it generates, e.g., images) in the adjusted position. Furthermore, the calibration data 502 can include stored perception sensor parameter values ​​that can be used by the calibration system 336 when generating a calibration transformation 338. For example, some perception sensor parameter values ​​(e.g., intrinsic parameter values) may be stored and used as default values ​​(and not recalculated). In another example, depending on the freedom of movement of the perception sensor 226, some extrinsic parameter values ​​may be stored and used as defaults.For example, if a perception sensor 226 is movable in the X, Y and Z directions, but the orientation (α, β and γ) cannot change, it may be that (α, β and γ) are stored and used as the default.

[0064] The machine data 503 comprises data indicating one or more machine properties of the working machines 100. The machine data 506 can include data specifying the type of machine (e.g., model, etc.), data specifying the machine's dimensions, data specifying the positions of machine components, a machine configuration (e.g., type and properties of attachments / working tools), data specifying machine operating values ​​(e.g., latency, etc.), and various other machine properties. The machine data 503 can include identifiers for the sensing sensors 226, as well as information about the dimensions, ranges of motion, mounting locations, initial or default position information, and various other information relating to the sensing sensors 226.The machine data 503 can be derived from a variety of sources, including, but not limited to, information provided by the dealer or manufacturer, operator or user input, stored machine identification information, and a variety of other sources.

[0065] The deployment location data 504 includes data specifying deployment location attributes that are derived from sources other than the sensors 208, or that may have been derived from the sensors 208 during past (historical) operations. As mentioned above, some data need not be derived from the sensors 208. For example, some perception sensor motion criteria data may, but are not limited to, originate from other sources. Third-party providers may, for instance, provide environmental data (e.g., weather data). Furthermore, operators or users may input various data. Some data may also be extracted from historical data (e.g., data collections from previous deployments). For example, but are not limited to, some environmental data (e.g., obstacle data) may be derived from historical data. The historical data may originate from the same machines or from different machines.

[0066] Data processing systems process and use sensor data 501, calibration data 502, machine data 503, deployment data 504 and other data 510.

[0067] For example, the data processing systems 330 can process sensor data 501 to generate processed sensor data. The processed sensor data can include computer-readable values ​​that are usable (or readable) by other elements of the system 500. Furthermore, the data processing systems 330 can use calibration data 502 (e.g., calibration transformations) to generate calibrated sensor data, e.g., calibrated images (or values ​​thereof) based on images generated by the perception sensors 226 and corresponding calibration transformations.

[0068] The 330 data processing system can include various processors or processing functions, including image processors or processing functions, sensor signal processors or processing functions, filter processors or processing functions, categorization processors or processing functions, normalization processors or processing functions, aggregation processors or processing functions, color extraction processors or processing functions, analog-to-digital transformation processors or processing functions, other transformation processors or processing functions (e.g., lookup tables, equations, mathematical functions, models, etc.), as well as various other data processing processors or processing functions.

[0069] Although the data processing system 330 is shown separately from the sensors 208, it is understood that in some examples the sensors 208 may include components of the data processing system 330. For example, but not limited to, a perception sensor 226 may include an image processor or image processing functions.

[0070] It is understood, therefore, that the 330 data processing systems can, for example, convert analog signals into readable digital signals (or digital values). It is understood that the 330 data processing systems can, for example, process image data to extract values ​​(e.g., pixel values, etc.) and furthermore, convert the extracted values. It is understood that the 330 data processing systems can perform preprocessing and postprocessing. It is understood that the 330 data processing systems can perform various forms of aggregation of the extracted or converted values. It is understood that the 330 data processing systems can apply calibration transformations to sensor data (e.g., images, etc.) to produce calibrated sensor data (e.g., calibrated images, etc.).

[0071] The perception sensor motion command system 332 can determine, based on one or more data elements 205 / 305 or based on outputs from other elements of the system 215 (e.g., power identification by the power identification system 340), that a perception sensor 226 should be moved (to change the position of the perception sensor 226) and generate a perception sensor motion command (instructions) to move the corresponding perception sensor 226 (to change the position of the perception sensor 226). The perception sensor motion command system 332 can, for example, be based on perception sensor motion criteria detected by the perception sensor motion criteria sensors 228, such as...The perception sensor motion command system 332 determines that a perception sensor should be moved based on one or more of the following criteria detected by sensors 240: one environmental criterion detected by sensors 240, one machine state criterion detected by sensors 242, one power criterion detected by sensors 244, one user input criterion detected by sensors 246, or one other perception sensor motion criterion detected by sensors 247. In another example, the perception sensor motion command system 332 can determine that a perception sensor should be moved based on a power identification by the power identification system 340. The perception sensor motion command system 332 also generates perception sensor motion commands (instructions) that can be issued to the control system 214 (e.g., the controllers 235).360°) to control the perception sensor position actuators 254 so that they move (or change the position of) perception sensors 226. A movement command (instruction) can define a new position into which a perception sensor 226 is to be moved.

[0072] The calibration trigger identification system 334 can detect calibration triggers and produce an output indicating an identified calibration trigger based on one or more data elements 205 / 305, based on issued motion commands (instructions) generated by the perception sensor motion command system 332, based on control signals generated by the control system 214, or based on both. A calibration operation must be performed for a perception sensor 226 when the perception sensor 226 changes its position. Thus, the calibration trigger identification system 334 is configured to recognize data indicating a change in the position of a perception sensor 226 (calibration trigger data) as a calibration trigger. Calibration trigger data can include sensor data generated by the perception sensor motion sensors 227, commands (e.g., control signals) issued by the control system 214 (e.g., control signals), and the system's control system 214.B. the controllers 235), such as commands (e.g., control signals) provided to the perception sensor position actuators 254, motion commands (instructions) issued by the perception sensor motion command system 332, or user inputs (e.g., detected by the user input sensors 246) that command movements of the perception sensors 226. The calibration trigger identification system 334 generates a calibration trigger output (e.g., a signal) in response to the detection of a calibration trigger, indicating the detection of a calibration trigger and, in some examples, identifying the corresponding perception sensor(s) 226.

[0073] The calibration system 336 is capable of performing calibration operations in response to a change in the position (movement) of the sensing sensors 226, which can be indicated by a calibration trigger output generated by the calibration trigger identification system 334, in order to calibrate (or recalibrate) the sensing sensors 226 during the operation of a machine 100. The calibration system 336 can determine parameters (e.g., intrinsic parameters, extrinsic parameters, etc.) of a sensing sensor 226 in a new position using a calibration algorithm. For example, the calibration system 336 is capable of identifying extrinsic parameters of a sensing sensor 226 in a new position, including a three-dimensional position (X, Y, and Z positions) and an orientation (α, β, and γ).The calibration system 336 can thus, in some examples, identify a six-degree vector (X, Y, Z, α, β, and γ) for localizing the perception sensor 226 as extrinsic parameters. Furthermore, the calibration system 336 can determine the position (location and orientation) of the perception sensor 226 relative to a previous position (or a standard or initial position) of the perception sensor 226, or relative to the frame or coordinate system of the machine 100, or relative to the location or position of another element of the machine (e.g., relative to the location or position of another perception sensor 226, for example, in the case of LiDAR camera calibration or radar camera calibration).The sensor data generated by the geoposition sensors 203 (as well as the machine data 504) can be used to determine the location of the frame of the machine 100 or the coordinate system of the machine 100, as well as the position of the perception sensor 226 relative to the frame or the coordinate system.

[0074] Additionally, the calibration system 336 can determine intrinsic parameters of a perception sensor 226 in a new position. For perception sensors 226 in the form of cameras, the intrinsic parameters can be, for example, a camera constant (C) or an xy-scale difference. (M) (or focal length in the X direction (fx) and focal length in the Y direction (fy), a principal point (x) H , y H), and include a shear parameter (S). In one example, the calibration system 336 is able to generate a multi-degree-of-freedom transformation (e.g., eleven degrees of freedom) (e.g., a direct linear transformation (DLT)) that describes a model of a perception sensor (e.g., a camera). Furthermore, the calibration system 336 can identify various other parameters, such as other intrinsic parameters, e.g., distortion parameters (e.g., lens distortion parameters). In another example, for perception sensors 226 in the form of LiDAR sensors, the intrinsic parameters may include distance accuracy, angular resolution, laser beam divergence, field of view (FOV), time offset, and intensity correction.In another example, for perception sensors 226 in the form of radar sensors, the intrinsic parameters can include a distance resolution, an angular resolution, a Doppler resolution, a field of view (FOV) and a beamwidth.

[0075] Using the identified parameters, the calibration system 336 can generate a calibration transformation (e.g., a model, a matrix, etc.) that can be used to accurately map a point in the three-dimensional world (mapped by the perception sensor 226) onto a two-dimensional (2D) image plane by using an equation such as x = PX, where x is the 2D pixel coordinate, P is the matrix, and X is the 3D world coordinate. The matrix (P) comprises the parameters identified by the calibration system 336.

[0076] Thus, the calibration transformation 338 generated by the calibration system 336 can include one or more perception sensor parameters (or their values), such as extrinsic and intrinsic parameters. In an example, the calibration transformation 338 can therefore be a model or matrix that includes the various perception sensor parameters (or their values).

[0077] As described above, in some examples, some of the perception sensor parameters (extrinsic and / or intrinsic parameters) can be stored (e.g., as calibration data 502) and used as a default instead of being recalculated by the calibration system 336. To identify such parameters, the calibration system 336 obtains them from the calibration data 502 (e.g., by retrieving or receiving them).

[0078] It is obvious to a specialist that there are numerous algorithms for calibrating perception sensors, which will only be briefly discussed here. All of these different perception sensor calibration algorithms are conceivable and can be used by the calibration system 336. One example of a calibration algorithm is perspective N-point calibration, which can employ direct linear transformation (DLT), iterative closest point (ICP), and Levenberg-Marquardt optimization. Some calibration algorithms use a calibration object (e.g., an object with a known profile or pattern, such as a checkerboard pattern) in the imaged environment to determine parameters. In some examples, one or more calibration objects could be attached to the work machine 100 (e.g., to the harvester header 104 of the harvester 100-1) to be captured in images generated by the perception sensors 226.In some examples, instead of placing a calibration object on the machine 100, an object on the machine 100 could have known coordinates (in a coordinate system of the machine) and be recognized (e.g., imaged) by the sensor to identify the sensor's parameters. Some calibration algorithms use neither a calibration object nor a known object in the scene, such as a self-calibration algorithm. One example of self-calibration uses multiple images and sensor movement (e.g., visual odometry) to determine sensor parameters. Another example of a calibration algorithm is base-area calibration, which can be used to determine sensor parameters.

[0079] In some examples, the calibration system 336 can generate calibration commands (instructions) that are used by the control system 214 (e.g., the controllers 235) to control the working machine 100 in order to perform the calibration. For example, the calibration system 336 can generate calibration commands (instructions) that instruct a travel speed control and can be used by the control system 214 (e.g., the controllers 235) to generate commands (control signals) to control the drive subsystem 250 in order to control a travel speed of the working machine 100 (e.g., to set it to a specific speed or to bring it to a standstill). Additionally or alternatively, the calibration system 336 can generate calibration commands (instructions) that instruct a travel direction ... control the drive subsystem 250 in order to control the travel speed of the working machine 100 (e.g., to set it to a specific speed or to bring it to a standstill).The control system 235) can be used to generate commands (control signals) for controlling the steering subsystem 252 to control the direction of travel of the working machine 100 (e.g., to follow a specific path, such as a straight path). Additionally or alternatively, the calibration system 336 can generate calibration commands (instructions) that direct a component controller and can be used by the control system 214 (e.g., the controllers 235) to generate commands (control signals) for controlling actuators 256 to control a component of the working machine 100. In an example of a component controller, calibration commands can, for instance, instruct that a component (e.g., the harvester header 104) of the working machine 100 (e.g., the harvester 100-1) be brought into a specific setting (e.g., raised to its maximum height) to be mapped as part of the calibration process.

[0080] It is understood that the calibration system 336 can use various data from data 205 / 305 when performing the calibration, for example, but not limited to, geographic geoposition sensor data generated by the geoposition sensors 203.

[0081] It is also understood that the calibration system 336 can use several calibration algorithms to determine parameters of a perception sensor 226.

[0082] As can be seen, the calibration system 336 is capable of performing a calibration and generating a calibration transformation 338 in response to a change in the position of a perception sensor 226. The calibration transformation 338 can be used by data processing systems 330 (at least some of which may be part of the perception sensors 226 in some examples, as described above) to calibrate the perception sensors 226 or to generate calibrated perception sensor data (e.g., calibrated images).

[0083] The power identification system 340 can identify the power and any change in power of a working machine 100 based on one or more data elements 205 / 305, such as sensor data 501 (e.g., sensor data generated by the power sensors 244). For example, if the power of the working machine 100 is below a desired level (e.g., target, threshold, etc.), it may be desirable to move a perception sensor 226, especially if the perception sensor 226 provides sensor data that is used in the control system and affects power. After a perception sensor 226 has been moved, the power identification system 340 can also identify the power of the working machine and compare the power after the movement with the power before the movement (or with a desired value).For example, if moving a perception sensor 226 does not result in better or worse performance than before the movement, it may be desirable to move the perception sensor 226 again. If moving a perception sensor 226 results in better performance than before the movement, but still does not reach a desired level, it may be desirable to move the perception sensor 226 again. In other examples, if a working machine is performing well, it may not be desirable to move a perception sensor 226, even if other perception movement criteria might indicate a need to move the perception sensor 226. The performance identification system 340 is capable of generating performance identification outputs that indicate identified performance and performance comparisons.The power identification outputs can be provided to or used by other elements of the System 500, including, but not limited to, the Perception Sensor Motion Command System 332, which can command a movement of the Perception Sensors 226 based on the power identification outputs.

[0084] The presentation generator system 342 can be operated to generate one or more presentations (e.g., visual, auditory, haptic, etc.) for display (e.g., display, auditory presentation, haptic presentation, etc.) on one or more interface mechanisms (e.g., 218 and / or 364). The presentations can, for example, inform an operator that a perception sensor 226 has changed its position, that calibration (recalibration) is complete, that a perception sensor 226 should be moved (change its position), and various other information. The presentations can, for example, display calibrated image data.

[0085] It is evident that the System 215 is capable of generating one or more dynamic Perception Monitoring Outputs 360 (hereinafter also referred to as Output / Outputs 360). An Output 360 may include one or more Perception Sensor Motion Commands (Instructions) generated by the Perception Sensor Motion Command System 332, Calibration Commands (Instructions) generated by the Calibration System 336, Calibrated Perception Sensor Data (Calibrated Images) generated by the Data Processing Systems 330, Presentations generated by the Presentation Generator 342, and various other elements or information generated by the System 215. An Output 360 may be used to control a work machine 100. For example, an Output 360 may be obtained (e.g., retrieved or received) from one or more Control Systems 214 to control a work machine 100, such as…by controlling one or more controllable subsystems 216 and / or one or more interface mechanisms 218 (e.g., for displaying information about (or based on) the output 360). Additionally or alternatively, an output 360 can be obtained (e.g., retrieved or received) by various other elements and used in various other ways. For example, but not limited to, a harvest logistics output 360 can be obtained (e.g., retrieved or received) by one or more other elements 362, such as by one or more interface mechanisms 364 (e.g., to present information about (or based on) the output 360, e.g., displaying information, etc.).

[0086] Fig. 6A and Fig. 6B (presented jointly as Fig. Figure 6) shows a flowchart illustrating an exemplary operation 700 of system 500 (e.g., the dynamic perception monitoring system 215) during the performance of dynamic perception sensor monitoring, calibration, and control based thereon. Fig. 6. It is assumed that the work machine 100 is in operation (e.g., at least switched on and also located at the place of use and able to move (e.g., drive)).

[0087] In Block 702, one or more data elements are obtained (e.g., retrieved or received) by System 500 (e.g., the Dynamic Perception Monitoring System 215). The obtained data may include sensor data 501, as indicated by Block 704. The obtained data may include calibration data 502, as indicated by Block 706. The obtained data may include machine data 503, as indicated by Block 708. The obtained data may include deployment location data 504, as indicated by Block 710. The obtained data may include various other data 510, as indicated by Block 711. Furthermore, the obtained data may include performance identification outputs generated by System 215 (e.g., the Performance Identification System 340). It is also understood that one or more of the data can be continuously obtained (or updated) via Process 700.

[0088] In block 713, system 215 (e.g., the perception sensor motion command system 332) determines, at least on the basis of the data obtained in block 702, whether the position of a perception sensor 226 should be changed. Some examples of determining whether the position of a perception sensor 226 should be changed have already been discussed above, in particular with reference to Fig. 5. For example, system 215 (e.g., the perception sensor motion command system 332) can determine whether the position of perception sensor 226 should be changed based on one or more perception sensor motion criteria contained in data 205 / 305 (e.g., sensor data 501, deployment location data 504, etc.), or based on the power identification outputs generated by the power identification system 340, or based on both. If block 713 determines that no change in position should occur, processing returns to block 702 or proceeds to block 736. If block 713 determines that a change in the position of one or more perception sensors 226 should occur, processing proceeds to block 714.

[0089] In block 714, system 215 (e.g., the perception sensor motion command system 332) generates perception sensor motion commands (instructions) which are provided to control system 214 (e.g., the controllers 235). Control system 214 (e.g., the controllers 235) generates commands (e.g., control signals) to control the perception sensor position actuators 254 so that they move a perception sensor 226 based on the perception sensor motion commands (instructions) (to change its position), as specified in block 715.

[0090] In block 716, system 215 (e.g., calibration trigger identification system 334) identifies a calibration trigger. As discussed previously, particularly with reference to Fig. 4 and Fig. 5, the system 215 (e.g., the calibration trigger identification system 334) can identify a calibration trigger based on data indicating a movement (change in position) of a perception sensor 226 (e.g., 205 / 305 (such as sensor data 504 (e.g., from the perception sensor motion sensors 227 or user input sensors 246), from perception sensor motion commands (instructions) generated by the perception sensor motion command system 332, or from commands (control signals) generated by the control system 214), as specified in block 718.

[0091] In block 718, in response to the identified calibration trigger, system 215 (e.g., calibration system 336) performs a calibration process to calibrate (or recalibrate) the perception sensors 226. Some examples of the operation of system 215 (e.g., calibration system 336) are given in connection with Fig. 5 explained. As stated in Block 719, the system 215 (e.g., the calibration system 336) can identify one or more perception sensor parameters corresponding to a perception sensor 226 in a new position, such as intrinsic parameters, extrinsic parameters, or a combination thereof. To identify the one or more perception sensor parameters, the system 215 (e.g., the calibration system 336) executes one or more calibration algorithms, as stated in Block 720. Some examples of calibration algorithms are given in connection with Fig. 5 discussed. In some examples, the system 215 (e.g., the calibration system 336) can generate calibration commands (instructions) that can be used by the control system 214 (e.g., the controllers 235) to control the working machine 100 to perform the calibration operation as specified in block 721. Some examples of calibration commands (instructions) and the associated machine control are discussed in connection with Fig. 5 explained.

[0092] System 215 (e.g., calibration system 336) thus generates a calibration transformation (e.g., 338). The calibration transformation can include one or more perception sensor parameters (or their values), such as extrinsic and intrinsic parameters. Therefore, in one example, the calibration transformation can be a model or matrix that includes the various perception sensor parameters (or their values). In some examples, as described above, some perception sensor parameters (or their values) can be stored in the calibration data 502 and used as a standard, and can therefore be identified by System 215 by obtaining the calibration data 502.

[0093] In Block 722, System 500 (e.g., Control System 214) controls the working machine 100, at least on the basis of perception sensor data generated by the perception sensor 226 in the new position and the calibration transformation. In one example, System 500 (e.g., System 215 (e.g., Data Processing Systems 330)) generates calibrated perception sensor data (e.g., calibrated images) on the basis of the calibration transformation, as specified in Block 724. The calibrated sensor data can be used to control the working machine 100. Controlling the working machine 100 can include controlling one or more controllable subsystems 216, as specified in Block 726. Controlling the working machine can additionally or alternatively include controlling one or more interface mechanisms 218, as specified in Block 728. The working machine 100 can also be controlled in other ways, as specified by Block 730.

[0094] In block 732, the system 215 (e.g., the power identification system 340) identifies the power output of the working machine 100 after movement (change of position) of the perception sensor 226 based on one or more data elements (e.g., data 205 / 305 (e.g., sensor data 501, such as power sensor data generated by the power sensors 244)). In block 734, the system 215 (e.g., the power identification system 340) determines whether the power output of the working machine 100 after the movement (change of position) of the perception sensor 226 is satisfactory (e.g., by comparison with the power output before the movement, by comparison with a threshold value or a target power output, or both). Some examples of the operation of the power identification system 340 are given in connection with Fig. 5 discussed.

[0095] If block 734 determines that the performance of the working machine 100 is unsatisfactory, processing returns to block 713, where it is determined whether the position of the perception sensor 226 should be changed again. If block 734 determines that the performance of the working machine 100 is satisfactory, processing continues in block 736.

[0096] Block 736 determines whether the operation of machine 100 is complete. If the operation of machine 100 is complete, processing returns to block 702. If the operation of machine 100 is complete in block 736, processing is terminated.

[0097] This discussion mentions processors and servers. In some examples, the processors and servers include computer processors with associated memory and timing circuitry, not shown separately. They are functional parts of the systems or devices to which they belong and are activated or enabled by the other components or elements in those systems.

[0098] Furthermore, various user interface displays were discussed. These displays can take a wide variety of forms and can incorporate a wide variety of user-operated interface mechanisms. For example, text fields, checkboxes, toggle buttons, links, drop-down menus, search fields, and so on can be among the user-activated interface mechanisms. These user-activated interface mechanisms can also be operated in a variety of other ways. For example, they can be operated using interface mechanisms such as a point-and-click device like a trackball or mouse, hardware buttons, switches, a joystick or keyboard, thumb switches or thumbpads, a virtual keyboard, or other virtual actuators.If the screen displaying the user interface mechanisms is a touchscreen, the user interface mechanisms can also be operated using touch gestures. Furthermore, the user interface mechanisms can be operated using voice commands via speech recognition functionality. Speech recognition can be implemented using a speech recognition device, such as a microphone, and software that recognizes the detected speech and executes commands based on the received speech.

[0099] Furthermore, various data storage devices were discussed. It should be noted that each data storage device can be subdivided into multiple data stores. In some examples, one or more of the data stores may be located locally to the systems accessing the data stores, one or more of the data stores may all be located remotely from a system using the data storage device, or one or more of the data stores may be located locally while others are located remotely. All of these configurations are considered by the present disclosure.

[0100] Furthermore, the figures depict a number of blocks, each assigned a specific functionality. It should be noted that fewer blocks can be used to illustrate that the functionality assigned to several different blocks is performed by fewer components. Conversely, more blocks can be used to illustrate that the functionality can be distributed across more components. In other examples, functions can be added and some removed.

[0101] It should be noted that the preceding discussion has described a variety of different systems, generators, controllers, components, and interactions. It is understood that any or all of such systems, generators, controllers, components, and interactions can be implemented by hardware elements, such as one or more processors, one or more processors that execute computer-executable instructions stored in memory, memory, or other processing components, some of which are described below, that perform the functions associated with these systems, generators, controllers, components, or interactions.Furthermore, some or all of the systems, generators, control devices, components, and interactions can be implemented by software that is loaded into memory and subsequently executed by one or more processors, servers, or other computing component(s), as described below. Any or all of the systems, generators, control devices, components, and interactions can also be implemented by various combinations of hardware, software, firmware, etc., some examples of which are described below. These are merely some examples of different structures that can be used to implement any or all of the systems, generators, control devices, components, and interactions described above. Other structures may also be used.

[0102] Fig. Figure 7 is a block diagram of a remote server architecture 1000. Fig. Figure 7 also shows one or more worker machines 100, one or more remote computing systems 300, and one or more remote user interface mechanisms 364 communicating with the remote server environment. The worker machines 100, the remote computing systems 300, and the remote user interface mechanisms 364 communicate with elements of a remote server architecture 1000. In some examples, the remote server architecture 1000 provides computing, software, data access, and storage services that do not require the end user to have any knowledge of the physical location or configuration of the system providing the services. In other examples, remote servers can provide the services over a wide area network, such as the internet, using appropriate protocols.For example, remote servers can provide applications over a wide area network that can be accessed via a web browser or any other computing device. The software or components shown in previous figures, as well as the associated data, can be stored on servers at a remote location. The computing resources in a remote server environment can be concentrated in a remote data center location, or they can be distributed across multiple remote data centers. Remote server infrastructures can deliver services across shared data centers, although the services appear as a single access point to the user. Thus, the components and functions described here can be provided by a remote server at a remote location using a remote server architecture.Alternatively, the components and functions can be provided by a server, or the components and functions can be installed directly, or in some other way, on client devices.

[0103] In the Fig. In the example shown, some elements resemble those shown in previous figures, and these elements are similarly numbered. Fig. Figure 7 shows in particular that the dynamic perception monitoring system 215, the data storage 204 or data storage 304, or a combination thereof, can be located at a server location 1002, which is remote from the work machines 100, the remote computing systems 300, and the remote user interface mechanisms 364. Therefore, in the Fig. In example 7, the work machines 100, the remote computing systems 300, and the interface mechanisms for remote users 364 access the systems via the remote server location 1002. In other examples, various other elements may also be located at server location 1002, such as various other elements of the system architecture 500.

[0104] Fig. Figure 7 also shows another example of a remote server architecture. Fig. Figure 7 shows that some elements from previous figures may be located at a remote server location 1002, while others may be located elsewhere. For example, one or more of the data stores 204 or 304 may be located at a location separate from location 1002, and storage access may be via the remote server at location 1002. Similarly, the dynamic perception monitoring system 215 may be located at a location separate from location 1002, and access may be via the remote server at location 1002. Regardless of where the elements are located, access to the elements may be directly by worker machines 100, remote computing systems 300, and remote user interface mechanisms 364 via a network, such as...a wide area network or a local area network; the elements can be hosted at a remote location by a service; or the elements can be provided as a service or be accessible through a connectivity service located at a remote location. Furthermore, data can be stored at any location, and the stored data can be accessible to or routed to operators, users, or systems. For example, physical carriers can be used instead of, or in addition to, carriers of electromagnetic waves. In some examples where wireless telecommunications service coverage is poor or nonexistent, another machine, such as a tanker truck or other mobile machine or vehicle, can have an automated, semi-automated, or manual information gathering system. If a mobile machine (e.g.,When the mobile machine (e.g., the mobile machine 100) approaches the machine containing the information collection system, such as a tanker truck before refueling, the information collection system gathers information from the mobile machine (e.g., the mobile machine 100) via any type of ad-hoc wireless connection. The gathered information can then be forwarded to another network when the machine containing the received information reaches a location where wireless telecommunications service or other wireless connectivity is available. For example, a tanker truck can enter an area with wireless communication coverage when traveling to a site to refuel other machines or when it is at a main fuel storage site. Other mobile machines or vehicles can enter an area with wireless communication coverage when traveling to other sites or when they are at a different location.All these architectures are included here. Furthermore, the information can be stored on a mobile machine (e.g., on work machine 100) until the mobile machine enters an area with wireless communication coverage. The mobile machine (e.g., work machine 100) can then send the information to another network.

[0105] It should also be noted that the elements of previous figures, or parts thereof, may be arranged on a wide variety of different devices. One or more of these devices may include an onboard computer, an electronic control unit, a display unit, a server, a desktop computer, a laptop computer, a tablet computer, or other mobile device, such as a palmtop computer, a mobile phone, a smartphone, a multimedia player, a personal digital assistant, etc.

[0106] In some examples, the remote server architecture can incorporate 1000 cybersecurity measures. Without limitation, these measures can include encrypting data on storage devices, encrypting data transmitted between network nodes, authenticating individuals or processes accessing data, and using ledgers to record metadata, data, data transfers, data access, and data transformations. In some examples, the ledgers can be distributed and immutable (e.g., implemented as a blockchain).

[0107] Fig. Figure 8 is a simplified block diagram of an illustrative example of a portable or mobile computing device that can be used as a portable device 16 by a user or client and in which the present system (or parts thereof) can be deployed. A mobile device can, for example, be deployed in the operator compartment of a mobile machine (e.g., work machine 100) or can be coupled to a mobile machine (e.g., work machine 100) via communication technology for use in generating, processing, or displaying the outputs discussed above (e.g., 360). Fig. 9 and Fig. 10 are examples of portable or mobile devices.

[0108] Fig. Figure 8 shows a general block diagram of the components of a client device 16, which can execute some of the components shown in the preceding figures, interact with them, or both. The device 16 provides a communication link 13 that enables the handheld device to communicate with other computing devices and, in some examples, provides a channel for automatically receiving information, such as by scanning. Examples of the communication link 13 include enabling communication via one or more communication protocols, such as wireless services used to provide cellular access to a network, and protocols that provide local wireless connections to networks.

[0109] In other examples, applications can be received on a removable Secure Digital (SD) card connected to a user interface 15. The interface 15 and communication links 13 communicate with a processor 17 (which can also represent processors or servers from other figures) along a bus 19, which is also connected to memory 21 and input / output (I / O) components 23, as well as a clock 25 and a location system 27.

[0110] The I / O components 23 are provided in an example to enable input and output operations. The I / O components 23 for various examples of the device 16 can include input components such as buttons, touch sensors, optical sensors, microphones, touchscreens, proximity sensors, accelerometers, orientation sensors, and output components such as a display device, a speaker, and / or a printer port. Other I / O components 23 can also be used.

[0111] Clock 25 includes, for example, a real-time clock component that outputs a time and date. This can also, for example, provide timing functions for processor 17.

[0112] The tracking system 27 includes, for example, a component that outputs a current geographic location of the device 16. This can be, for example, a GPS (Global Positioning System) receiver, a LORAN system, a dead reckoning navigation system, a cellular triangulation system, or another positioning system. The tracking system 27 can also include, for example, mapping software or navigation software that generates desired maps, navigation routes, and other geographic functions.

[0113] Memory 21 stores an operating system 29, network settings 31, applications 33, application configuration settings 35, a client system 24, a data store 37, communication drivers 39, and communication configuration settings 41. Memory 21 can include all types of physical volatile and non-volatile computer-readable storage devices. Memory 21 can also include computer storage media (described below). Memory 21 stores computer-readable instructions which, when executed by the processor 17, cause the processor to perform computer-implemented steps or functions according to the instructions. The processor 17 can be activated by other components to enable their functionality as well.

[0114] Fig. Figure 9 shows an example where the device 16 is a tablet computer 1100. Fig. Figure 9 shows the Computer 1100 with a user interface display screen 1102. The screen 1102 can be a touchscreen or a pen-enabled user interface that receives input from a pen or stylus. The Tablet Computer 1100 can also use a virtual keyboard on the screen. Naturally, the Computer 1100 can also be attached to a keyboard or other user input device via a suitable mounting mechanism, such as a wireless link or a USB connection. The Computer 1100 can also receive voice input, for example.

[0115] Fig. 10 resembles Fig. 10 except that the device is a smartphone 71. The smartphone 71 has a touch-sensitive display 73 that shows icons or tiles or other user input mechanisms 75. The mechanisms 75 can be used by a user to run applications, make calls, perform data transfer operations, etc. In general, the smartphone 71 is built on a mobile operating system and offers more advanced computing capabilities and connectivity than a feature phone.

[0116] It should be noted that other forms of devices 16 are possible.

[0117] Fig. Figure 11 is an example of a computational environment in which elements from previously described figures can be used. With reference to Fig. Figure 11 comprises an example system for implementing some embodiments, comprising a computing device in the form of a computer 1210 programmed to operate as described above. The components of the computer 1210 may, but are not limited to, include a processing unit 1220 (which may include processors or servers from previous figures), a system memory 1230, and a system bus 1221 connecting various system components, including the system memory, to the processing unit 1220. The system bus 1221 may be one of several types of bus structure, including a memory bus or memory controller, a peripheral bus, and a local bus, employing any variety of bus architectures. The memory and programs described with reference to the preceding figures may be represented in corresponding parts of Fig. 11 will be used.

[0118] The Computer 1210 typically incorporates a variety of computer-readable media. Computer-readable media can be any available media that the Computer 1210 can access, including both volatile and non-volatile, removable and non-removable media. By way of example, and without limitation, computer-readable media can include computer storage media and communication media. Computer storage media are distinct from and do not include a modulated data signal or carrier wave. Computer-readable media include hardware storage media, including both volatile and non-volatile, removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, DVD (Digital Versatile Discs) or other optical disc storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that the Computer 1210 can access. Communication media can embody computer-readable instructions, data structures, program modules, or other data in a transport mechanism and include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or modified in such a way that information is encoded in the signal.

[0119] System memory 1230 comprises computer storage media in the form of volatile and / or non-volatile memory, or both, such as ROM (read-only memory) 1231 and RAM (random access memory) 1232. A BIOS (basic input / output system) 1233, which contains the basic routines that assist in transferring information between elements within the computer 1210, such as during startup, is typically stored in ROM 1231. RAM 1232 typically contains data and / or program modules, or both, that are immediately accessible and / or are currently being processed by the processing unit 1220. This is illustrated by way of example and without limitation. Fig. 11 an operating system 1234, application programs 1235, other program modules 1236 and program data 1237.

[0120] The Computer 1210 may also include other removable / non-removable volatile / non-volatile computer storage media. These are merely examples. Fig. 11 a hard disk drive 1241, which reads from or writes to the non-removable non-volatile magnetic media, an optical disk drive 1255 and a non-volatile optical disk 1256. The hard disk drive 1241 is typically connected to the system bus 1221 via a non-removable storage interface, such as the 1240 interface, and the optical disk drive 1255 is typically connected to the system bus 1221 via a removable storage interface, such as the 1250 interface.

[0121] Alternatively or additionally, the functionality described here can be implemented, at least partially, by one or more hardware logic components. Examples of suitable hardware logic components include, but are not limited to, freely programmable gate arrays (FPGAs), application-specific integrated circuits (e.g., ASICs), application-specific standard products (e.g., ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), quantum computers, etc.

[0122] The drives and their aforementioned and in Fig. The 11 illustrated associated computer storage media provide storage for computer-readable instructions, data structures, program modules, and other data for the Computer 1210. Fig. Figure 11, for example, illustrates the hard disk drive 1241 as storing the operating system 1244, the application programs 1245, other program modules 1246, and the program data 1247. It should be noted that these components can either be the same as the operating system 1234, the application programs 1235, the other program modules 1236, and the program data 1237, or they can be different from them.

[0123] A user can input commands and information into the computer 1210 via input devices such as a keyboard 1262, a microphone 1263, and a pointing device 1261, such as a mouse, trackball, or touchpad. Other input devices (not shown) may include a joystick, gamepad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 1220 via a user input interface 1260 coupled to the system bus, but may also be connected via other interface and bus structures. A visual display 1291 or other type of display device is also connected to the system bus 1221 via an interface such as a video interface 1290.In addition to the monitor, computers may also include other peripheral output devices, such as loudspeakers 1297 and a printer 1296, which may be connected via an output peripheral interface 1295.

[0124] The computer 1210 is operated in a networked environment via logical connections (e.g. Controller Area Network - CAN, Local Area Network - LAN or Wide Area Network - WAN) with one or more remote computers, such as a remote computer 1280.

[0125] When used in a LAN network environment, the computer 1210 is connected to the LAN 1271 via a network interface or adapter 1270. When used in a WAN networking environment, the computer 1210 typically includes a modem 1272 or other means of establishing communications over the WAN 1273, such as the Internet. In a networked environment, program modules can be stored in a remote storage device. Fig. For example, 11 shows that remote application programs 1285 can be located on the remote computer 1280.

[0126] It should also be noted that the various examples described here can be combined in different ways. That is, parts of one or more examples can be combined with parts of one or more other examples. All of this is taken into consideration here.

[0127] Even if the subject matter is described in a language specific to structural features and / or methodological actions, it is understood that the subject matter defined in the accompanying claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as examples of the claims.

Claims

[1] System (500) comprising: a perception sensor (226) associated with a working machine (100) and configured to capture an image that displays one or more properties; a perception position actuator (254) that is controllable to change the position of the perception sensor during operation of the working machine; one or more processors (201, 301); and a memory (204, 304) that stores instructions (205, 305) that can be executed by the one or more processors and that, when executed by the one or more processors, cause the one or more processors to: during the operation of the working machine and in response to a calibration trigger (716), perform a calibration process (718) to generate a calibration transformation (338); and control the working machine at least on the basis of the calibration transformation and the image (722). [2] System according to claim 1, wherein the instructions, when executed by the one or more processors, cause the one or more processors to identify one or more parameters of the perception sensor based on the calibration process (719). [3] System according to claim 1, wherein the one or more parameters comprise one or more of the following: (i) one or more extrinsic parameters; or (ii) one or more intrinsic parameters. [4] System according to claim 1, wherein the calibration trigger indicates a change in the position of the perception sensor (717). [5] System according to claim 1, wherein the instructions, when executed by the one or more processors, cause the one or more processors to control the perception position actuator based on criteria to change the position of the perception sensor. [6] System according to claim 5, wherein the criteria comprise one or more of the following: (i) environmental data; (ii) machine state data; (iii) machine performance data; or (iv) user input data. [7] System according to claim 1, wherein the instructions, when executed by the one or more processors, cause the one or more processors to control the working machine by controlling one or more controllable subsystems (216) of the working machine. [8] System according to claim 1, wherein the instructions, when executed by the one or more processors, cause the one or more processors to control the working machine by controlling one or more interface mechanisms (218) of the working machine. [9] System according to claim 1, wherein the working machine comprises an agricultural harvesting vehicle (100-1). [10] Computer-implemented method for controlling a working machine (100), wherein the computer-implemented method comprises: Detection (716) of a calibration trigger associated with a perception sensor (226) corresponding to the working machine; Performing (718) a calibration operation during the operation of the working machine and in response to the detected calibration trigger to generate a calibration transformation (338); and Control (722) the working machine at least on the basis of the calibration transformation. [11] Computer-implemented method according to claim 10, further comprising identifying (719) one or more parameters of the perception sensor based on the calibration process. [12] Computer-implemented method according to claim 11, wherein identifying one or more parameters of the perception sensor comprises identifying one or more of the following: (i) one or more extrinsic parameters of the perception sensor; or (ii) one or more intrinsic parameters of the perception sensor. [13] Computer-implemented method according to claim 10, wherein the detection of the calibration trigger comprises receiving data (716) indicating a change in position of the sensing sensor. [14] Computer-implemented method according to claim 10, wherein controlling the working machine comprises controlling one or more controllable subsystems (216) of the working machine. [15] Computer-implemented method according to claim 10, wherein controlling the working machine comprises controlling one or more interface mechanisms (218) of the working machine.

Images