Excluding machine parts from video frames based on motion information

Abstract

A controller may process a plurality of video frames to determine apparent motion of each pixel of one or more pixels or each pixel group of one or more pixel groups of each video frame of the plurality of video frames. The controller may select one or more processed video frames corresponding to a duration of time from the plurality of processed video frames, and may generate a composite video frame based on the one or more processed video frames. The controller may generate a video frame mask based on the composite video frame, and may obtain additional video data including at least one additional video frame. The controller may cause the video frame mask to be applied to the at least one additional video frame, and may process the at least one additional video frame using an object detection technique.

Description

Technical Field

[0001] The present invention generally relates to processing video data and to a controller for processing video data to exclude components of a work machine from a video frame based on motion information. Background Art

[0002] In various work industries (e.g., construction, material moving, mining, hauling, etc.), objects near or in the path of a work machine on a work site may interfere with the work operation of the work machine. Furthermore, if these objects come into contact with the work machine, these objects may cause damage to the work machine or other work machines operating on the work site. Therefore, object detection is often used to ensure that the operator of the work machine is aware of the presence of objects located near the work machine on the work site where the work machine is operating.

[0003] An attempt to monitor the surrounding environment of a working machine is disclosed in U.S. Patent Application Publication No. 2018 / 0209122 (“the '122 Publication”) filed by Sumitomo Heavy Industries, Ltd. on March 19, 2018. In particular, the '122 Publication discloses a controller that receives the output of an image capture device and executes a software program to correspond to an extraction portion and a recognition portion of the controller. According to the '122 Publication text, the extraction portion extracts a recognition processing target image, which is a portion of an image (e.g., a portion of a captured image) to be subjected to subsequent image processing. As disclosed in the '122 Publication, the extraction portion masks a recognition processing unsuitable area of ​​a target image area of ​​the target image (e.g., an area in which the body of the shovel of the working machine is captured) to generate a normalized image of the target image. Therefore, according to the '122 Publication, by masking the image of the recognition processing unsuitable area, the extraction portion prevents the image of the recognition processing unsuitable area from affecting the recognition processing of the recognition portion. According to the '122 publication, this enables the recognition section to use the image of the area other than the mask area in the normalized image to recognize whether it is a person image without being affected by the image of the area inappropriate for recognition processing.

[0004] Although the controller of the '122 publication can mask the unsuitable identification area of ​​the target image, the '122 publication does not disclose how the controller identifies the unsuitable identification area. In addition, the '122 publication does not address the identification area associated with the mobile implement (e.g., shovel) of the work machine in the target image stream.

[0005] The controller of the present invention solves one or more of the problems set forth above and / or other problems in the art. Summary of the invention

[0006] According to some implementations, a method may include obtaining, by a device, video data comprising a plurality of video frames; processing, by the device, the plurality of video frames to determine the apparent motion of each pixel of one or more pixels or each pixel group of one or more pixel groups of each video frame of the plurality of video frames; selecting, by the device, one or more processed video frames corresponding to a duration from the plurality of processed video frames; generating, by the device, a composite video frame based on the one or more processed video frames; generating, by the device, a video frame mask based on the composite video frame; obtaining, by the device, additional video data comprising at least one additional video frame; applying, by the device, the video frame mask to the at least one additional video frame; and processing, by the device, the at least one additional video frame using an object detection image processing technique and based on applying the video frame mask to the at least one additional video frame.

[0007] According to some implementations, a device may include one or more memories; and one or more processors communicatively coupled to the one or more memories, configured to: one or more processors communicatively coupled to the one or more memories to: obtain video data including multiple video frames; process the multiple video frames to determine the apparent motion of each pixel of one or more pixels or each pixel group of one or more pixel groups of each video frame of the multiple video frames; select one or more processed video frames corresponding to a duration in the multiple processed video frames; generate a composite video frame based on the one or more processed video frames; determine at least one area of ​​the composite video frame associated with at least one component of a working machine; generate a video frame mask based on the composite video frame and the at least one area of ​​the composite video frame; obtain additional video data including at least one additional video frame; apply the video frame mask to the at least one additional video frame; and process the at least one additional video frame using object detection image processing technology and based on applying the video frame mask to the at least one additional video frame.

[0008] According to some implementations, a work machine may include an implement; a camera device; and a control device for: obtaining video data including a plurality of video frames from the camera device; processing the plurality of video frames to determine the apparent motion of each pixel of one or more pixels or each pixel group of one or more pixel groups of each video frame of the plurality of video frames; selecting one or more processed video frames corresponding to a duration from the plurality of processed video frames; generating a composite video frame based on the one or more processed video frames; obtaining one or more control instructions or sensor data related to the motion of the implement or the work machine; generating a video frame mask based on the composite video frame and the one or more control instructions or sensor data; obtaining additional video data including at least one additional video frame from the camera device; applying the video frame mask to the at least one additional video frame; and processing the at least one additional video frame using object detection image processing technology and based on applying the video frame mask to the at least one additional video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram of an example work machine as described herein.

[0010] Figure 2 is a flow chart that graphically illustrates a process for excluding components of a work machine from a video frame based on motion information as described herein.

[0011] Figure 3 is a graph of an example video frame.

[0012] Figure 4 yes Figure 1 A diagram of example components of one or more devices.

[0013] Figure 5 is a flow chart of an example process for excluding components of a work machine from a video frame based on motion information as described herein. DETAILED DESCRIPTION

[0014] Figure 11 is a diagram of an example work machine 100 as described herein. The work machine 100 may include any machine that performs some type of operation associated with an industry such as mining, construction, farming, etc. For example, the work machine 100 may be an earth moving machine, such as a bulldozer, a loader, a backhoe, an excavator, a motor grader, or any other earth moving machine. The work machine 100 may traverse a work site to manipulate materials below a working surface 102 (e.g., transport, cultivate, dig, tear, etc.). The work machine 100 may include a power source 104 configured to generate mechanical power, a traction device 106, at least one implement 108 (e.g., at least one material engaging element, such as a ripper), and an operator station 110 that houses operator controls. The work machine 100 may also include a frame 112 configured to support one or more components of the work machine 100.

[0015] The power source 104 may be any type of internal combustion engine, such as a diesel engine, a gasoline engine, or a gas fuel powered engine. In addition, the power source 104 may be a non-engine type power generation device, such as a fuel cell, a battery, an electric motor, or other type of power source. The power source 104 may generate a variable power output directed to the implement 108 and the traction device 106 in response to one or more inputs.

[0016] The traction device 106 may include tracks located on each side of the work machine 100 (only one side is shown) and operably driven by one or more sprockets 114. The sprockets 114 are operably connected to the power source 104 to receive power therefrom and drive the traction device 106. The movement of the traction device 106 can propel the work machine 100 relative to the work surface 102. The traction device 106 may additionally or alternatively include wheels, belts, or other traction devices, which may or may not be steerable. The traction device 106 may be hydraulically actuated, mechanically actuated, electronically actuated, or actuated in any other suitable manner.

[0017] The implement 108 may be configured to be raised, lowered, and tilted relative to the frame 112. For example, when the implement 108 is a scarifier, the implement 108 may include a handle 116 held in place by a mounting member 118. The handle 116 may penetrate the working surface 102 to disturb or damage (e.g., tear) material below the working surface 102, and may be movable relative to the mounting member 118. The mounting member 118 may be connected to the frame 112 by a connection system and / or in any other suitable manner, wherein at least one implement actuator forms a member of the connection system. For example, a first hydraulic actuator 120 may be connected to raise and lower the implement 108, and a second hydraulic actuator 122 may be connected to tilt the implement 108. The implement 108 may alternatively include a plow, a tine, a cultivator, etc.

[0018] Hydraulic actuators 120 and 122 may each include a piston-cylinder arrangement, a hydraulic motor, and / or another known hydraulic device having one or more fluid chambers therein. Movement of hydraulic actuator 120 may help move implement 108 relative to frame 112 and work surface 102, such as downwardly toward and upwardly away from work surface 102. Extension of hydraulic actuator 120 may be related to the position of implement 108 relative to work surface 102. Similarly, extension of hydraulic actuator 122 may be related to the orientation of implement 108 relative to frame 112 and / or work surface 102, such as decreasing or increasing the angle of implement 108 relative to work surface 102.

[0019] The operator station 110 may provide a control interface for an operator of the work machine 100. For example, the operator station 110 may include an accelerator pedal 124, implement controls 126, and / or output devices 128. Although not shown, the operator station 110 may also include other controls, such as machine direction controls, a deceleration pedal, and / or any other similar controls. The accelerator pedal 124 may determine the amount of mechanical power delivered to the traction device 106 to move the traction device 106.

[0020] The implement control 126 may allow an operator of the work machine 100 to manipulate the implement 108. More specifically, the implement control 126 may control the amount or pressure of fluid supplied to and exhausted from the hydraulic actuators 120 and 122. Thus, the implement control 126 may allow the operator to set the height of the handle 116 above or below the work surface 102 and to set the angle of the handle 116 relative to the work surface 102. The implement control 126 may also allow the operator to change the angle of the handle 116 relative to the work surface 102 when the handle 116 is above or below the work surface 102.

[0021] Output device 128 may include an audio output device, a vibration output device, a visual display, etc. The audio output device may be any audio device capable of providing an audible signal to the operator of work machine 100, such as a speaker. The visual display may be a screen or any visual device that may be configured to provide any form of visual stimulation or visual feedback to the operator. Although the visual display is Figure 1 Although depicted as being within work machine 100, the visual display may be off-board (e.g., at a remote operator station) or in any location suitable for being viewed by an operator or any other user relative to work machine 100. Additionally, the vibro-output device may be any vibro- and / or haptic device configured to alert an operator via one or more vibrations.

[0022] The work machine 100 may include one or more sensing devices 130 (individually referred to as a "sensing device 130" and collectively referred to as "sensing devices 130"), a controller 132, and / or one or more sensors 134 ( Figure 1 The sensing device 130 may be operatively associated with the work machine 100 by, for example, mounting or otherwise attaching the sensing device 130 to the work machine 100. However, the sensing device 130 need not be attached to the work machine 100 and may be positioned relative to the work machine 100 in any manner in which the sensing device 130, in operation, may obtain sensory data associated with an area proximate to the work machine 100. The sensing device 130 may include a camera device, a radar device, a light detection and ranging (lidar device), a sonar device, and the like.

[0023] The sensors 134 may be operably associated with the work machine 100 and / or components of the work machine 100 by, for example, mounting or otherwise attaching the sensors 134 to the work machine 100 and / or components of the work machine 100. In one example, the sensors 134 may include a global positioning system (GPS) device, an inertial measurement unit (IMU) device, or the like, and may obtain sensor data regarding the location, position, location, orientation, acceleration, specific force, etc. of the machine 100 and / or components of the work machine 100. In additional examples, the sensors 134 may be associated with one or more sprockets 114 and / or traction devices 106 and may obtain sensor data regarding traction device or sprocket speed, traction device or sprocket slip, etc. In another example, the sensors 134 may be associated with the implement 108 and may obtain sensor data regarding the height, angle, torque, load, etc. of the implement 108.

[0024] Sensing device 130 may be configured to transmit sensed data to, for example, controller 132; however, sensing device 130 is not limited to transmitting sensed data only to controller 132, and may additionally or alternatively transmit sensed data to any additional or alternative controller (e.g., included on work machine 100, remotely located relative to work machine 100, etc.). In some implementations, sensing device 130 may include controller 132, or controller 132 may include sensing device 130. Similarly, sensor 134 may be configured to transmit sensor data to controller 132 or any additional or alternative controller.

[0025] The controller 132 may be configured to process the sensed data and / or the sensor data. The controller 132 may be implemented in hardware, firmware, or a combination of hardware and software. Further functions of the controller 132 are described herein. Figure 2 Give a description.

[0026] Figure 1 The number and arrangement of components shown are provided as examples. In practice, work machine 100 may include Figure 1 Additional components, fewer components, different components, or differently arranged components than those shown in .One set of components (eg, one or more components) of work machine 100 may perform one or more functions described as being performed by another set of components of work machine 100 .

[0027] Figure 2 is a flow chart that graphically illustrates a process 200 for excluding components of a work machine from a video frame based on motion information as described herein. In some implementations, Figure 2 One or more operations of may be performed by controller 132 alone or in combination with another device, such as output device 128, sensing device 130, or the like.

[0028] With respect to reference numeral 210, the controller 132 may obtain sensing data from the sensing device 130. The sensing data may be data associated with the type of the sensing device 130. For example, when the sensing device 130 is a camera device, the sensing data may be video data. Similarly, when the sensing device 130 is a radar device, a lidar device, a sonar device, etc., the sensing data may be radar data, lidar data, sonar data, etc. The sensing data may include one or more sensing elements (also referred to as frames). For example, the video data may include multiple video frames (e.g., a single video frame 215 is shown in conjunction with reference numeral 210), the radar data may include multiple radar frames, the lidar data may include multiple lidar frames, the sonar data may include multiple sonar frames, etc. Although some implementations as described herein involve operations associated with video data and video frames, implementations involving similar operations associated with other types of sensing data and / or sensing elements are also contemplated.

[0029] In conjunction with reference numeral 220, the controller 132 may process one or more video frames of the plurality of video frames (e.g., shown as one or more processed video frames 225). The controller 132 may process the video frames to determine the apparent motion of each of the one or more pixels of the video frame and / or the apparent motion of each pixel group of the one or more pixels of the video frame. The apparent motion of a pixel may include the magnitude and / or direction of the motion of the pixel compared to a corresponding pixel in another video frame. The magnitude may be the difference between the position of the pixel and the position of the corresponding pixel (e.g., measured using the horizontal coordinates and / or vertical coordinates of the pixel and the corresponding pixel), and the direction may be the position of the pixel relative to the position of the corresponding pixel (e.g., measured in degrees). Similarly, the apparent motion of a pixel group may include the magnitude and / or direction of the motion of the pixel group compared to a corresponding pixel group in another video frame. The magnitude may be the difference between the position of the pixel group and the position of the corresponding pixel group, and the direction may be the position of the pixel group relative to the position of the corresponding pixel group.

[0030] For example, the controller 132 may compare a first video frame and a second video frame (e.g., adjacent video frames) in one or more video frames to determine the apparent motion of at least one pixel, at least one pixel group, etc. in the second video frame. The controller 132 may identify at least one pixel, at least one pixel group, etc. in the second video frame and search for the corresponding at least one pixel, at least one pixel group, etc. in the first video frame (e.g., by searching for pixels, pixel groups, etc. in the first video frame that are associated with one or more similar parameters (e.g., pixel color value, pixel brightness value, pixel group color value, pixel group brightness value, etc.) as the at least one pixel, at least one pixel group, etc. in the second video frame). The controller 132 may compare the position of the at least one pixel, at least one pixel group, etc. in the second video frame with the position of the corresponding at least one pixel, at least one pixel group, etc. in the first video frame to determine the apparent motion of the at least one pixel, at least one pixel group, etc. in the second video frame (e.g., the magnitude and / or direction of the motion).

[0031] In some implementations, the controller 132 may process the video frame using an optical flow image processing technique (e.g., a dense optical flow image processing technique (e.g., a Lucas-Kanade image processing technique), a coarse optical flow image processing technique (e.g., a Black-Jepson image processing technique), etc.) to determine the apparent motion of one or more pixels of the video frame (e.g., the apparent motion of a pixel, the apparent motion of a pixel group, etc.). When using the optical flow image processing technique, the apparent motion of a pixel, a pixel group, etc. is also referred to as a flow component of the pixel, the pixel group, etc.

[0032] After determining the apparent motion of a pixel, pixel group, etc. of a video frame, the controller 132 may cause a change in a parameter associated with the pixel, pixel group, etc. (e.g., a pixel color value, a pixel brightness value, a pixel group color value, a pixel group brightness value, etc.). For example, the controller 132 may cause the pixel color value of at least one pixel to change to a value that represents the magnitude and / or direction of movement of the at least one pixel (e.g., dark green may indicate a high magnitude, moving to the left; light green may indicate a low magnitude, moving to the left; light red may indicate a low magnitude, moving to the right; dark red may indicate a high magnitude, moving to the right; etc.).

[0033] In some implementations, the controller 132 may select one or more of the processed video frames. For example, the controller 132 may select one or more processed video frames corresponding to a duration (e.g., the one or more processed video frames are sequentially generated during a one-second duration, a ten-second duration, a one-minute duration, etc.).

[0034] In conjunction with reference numeral 230, the controller 132 may generate a composite video frame (e.g., shown as composite video frame 235) based on the selected one or more processed video frames (e.g., one or more processed video frames corresponding to the duration). To generate the pixels, pixel groups, etc. of the composite video frame, the controller 132 may identify the respective corresponding pixels, respective corresponding pixel groups, etc. in each of the selected one or more processed video frames (e.g., where each corresponding pixel, each corresponding pixel group, etc. has the same respective position in each processed video frame as the position of the pixel, pixel group, etc. of the composite video frame). The controller 132 may identify the respective parameter values ​​(e.g., pixel color value, pixel brightness value, pixel group color value, pixel group brightness value, etc.) of each respective pixel, each pixel group, etc., and determine an average value of the respective parameter values. The controller 132 may cause the parameter values ​​of the pixels, pixel groups, etc. of the composite video to have a value equal to the average value. Thus, each pixel, each pixel group, etc. of the composite video frame indicates the average apparent motion (e.g., average magnitude and direction of motion) of the corresponding pixel, corresponding pixel group, etc. in the one or more processed video frames. In this manner, the composite video frame may indicate motion (eg, average apparent motion) associated with one or more processed video frames over a duration.

[0035] In conjunction with reference numeral 240, the controller 132 may determine at least one region of the composite video frame (eg, Figure 2100 ). The at least one region may be associated with a lack of motion (e.g., a corresponding parameter of each pixel, each pixel group, etc. of the at least one region has a value indicating little or no apparent motion). Thus, when at least one component of the work machine 100 (e.g., the implement 108, the frame 112, etc.) appears to be motionless relative to the environment in one or more processed video frames used to generate the composite video frame, the at least one region may be associated with the at least one component of the work machine 100. To determine at least one region of the composite video frame, the controller 132 may identify at least one pixel, at least one pixel group, etc. of the composite video frame that has a parameter value within a specific range, such as a pixel color value, a pixel group color, etc., the specific range being, for example, a specific color range (e.g., a range of color values ​​indicating little or no apparent motion of the pixel, pixel group, etc.), and may determine the at least one region of the composite video frame based on the parameter value of the at least one pixel, the at least one pixel group, etc.

[0036] Additionally, or alternatively, the controller 132 may obtain information about the movement of the working machine 100 (for example, one or more control instructions about the movement of the working machine 100 generated by the accelerator pedal 124 and / or the deceleration pedal, such as the speed or acceleration of the working machine 100; one or more control instructions about the movement of the working machine 100 generated by the machine direction control, such as the steering of the working machine 100; sensor data about the movement of the working machine 100 obtained by the sensor 134, such as the position, location, position, orientation, acceleration, specific force, etc. of the machine 100, etc.). In addition, the controller 132 may obtain information about the movement of at least one component of the work machine (e.g., one or more control instructions generated by the implement control 126 regarding the movement of the implement 108, such as a change in height or angle of the implement 108; one or more control instructions generated by the accelerator pedal 124 and / or the decelerator pedal regarding the movement of the traction device 106, such as a speed or acceleration of the traction device 106; sensor data obtained by the sensor 134 regarding the movement of one or more sprockets 114 and / or traction devices 106, such as a traction device or sprocket speed and / or traction device or sprocket slip; sensor data obtained by the sensor 134 regarding the movement of the implement 108, such as a change in height, angle, torque, load, etc. of the implement 108; etc.). In some implementations, the controller 132 may determine at least one region of the composite video frame based on the information about the movement of the work machine 100, the information about the movement of at least one component, etc. At least one area of ​​the composite video frame may include one or more pixels, one or more groups of pixels, etc., each of which indicates an apparent motion (e.g., amplitude and / or direction of motion) corresponding to information about the motion of the working machine 100, information about the motion of at least one component, etc.

[0037] For example, the controller 132 may determine the magnitude and direction of the movement of the at least one component based on the information about the movement of the at least one component (e.g., the implement 108 is being raised or lowered at a particular speed). The controller 132 may determine at least one region of the composite video frame by identifying one or more regions of the composite video frame having corresponding pixels, corresponding groups of pixels, etc. indicating an apparent movement (e.g., the magnitude and / or direction of the movement) corresponding to the magnitude and direction of the movement of the at least one component (e.g., the pixels, groups of pixels, etc. indicating the magnitude and direction of the movement associated with the implement 108 being raised or lowered at a particular speed). As another example, the controller 132 may determine the magnitude and direction of the movement of the work machine 100 based on the information about the movement of the work machine 100 (e.g., the direction and speed of the work machine 100), and may determine the magnitude and direction of the movement of the at least one component based on the information about the movement of the at least one component (e.g., the direction and speed of the traction device 106). The controller 132 may determine at least one region of the composite video frame by identifying one or more regions of the composite video frame having corresponding pixels, corresponding groups of pixels, etc. that indicate an apparent motion (e.g., the magnitude and / or direction of the motion) that corresponds to the magnitude and direction of the motion of the at least one component (e.g., pixels, groups of pixels, etc. that indicate the magnitude and direction of the motion that is related to the direction and speed of the traction device 106) and / or that does not correspond to the magnitude and direction of the motion of the work machine 100 (e.g., pixels, groups of pixels, etc. that indicate the magnitude and direction of the motion that is not related to the direction and speed of the work machine 100). Other examples of controller 132 determining at least one region are described herein with respect to Figure 3 Give a description.

[0038] With reference to numeral 250, the controller 132 may generate a video frame mask (e.g., in the example of FIG. 1 ) based on the composite video frame, at least one region of the composite video frame, information about the motion of the work machine 100, information about the motion of at least one component, etc. Figure 2 255). To generate the video frame mask, the controller 132 may generate a video frame including a plurality of pixels and, for each pixel, each pixel group, etc. of the plurality of pixels corresponding to at least one region of the composite video frame, associate a parameter of the pixel, pixel group, etc. with a particular value. For example, the controller 132 may associate a pixel color value of the pixel with a particular color (e.g., black). As another example, the controller 132 may associate a pixel brightness value of the pixel with a particular amount of brightness (e.g., no brightness).

[0039] In some embodiments, when generating the video frame mask, the controller 132 may ignore portions of the composite video frame. For example, the controller 132 may ignore a portion of the composite video frame associated with an upper portion of the composite video frame (e.g., ignore pixels, groups of pixels, etc. associated with the sky). Thus, in some implementations, to generate the video frame mask, the controller 132 may generate a video frame comprising a plurality of pixels, and in a manner similar to that described herein, for each pixel, each group of pixels, etc. of the plurality of pixels that correspond to at least one region of the composite video frame and do not correspond to the ignored portion of the composite video frame, associate a parameter of the pixel with a particular value.

[0040] In conjunction with reference numeral 260, the controller 132 may obtain additional video data from the sensing device 130. The additional video data may include at least one additional video frame (in Figure 2 Controller 132 may cause the video frame mask to be applied to at least one additional video frame (eg, combine at least one additional video frame and the video frame mask).

[0041] To apply the video frame mask to the at least one additional video frame, the controller 132 may identify at least one pixel, at least one pixel group, etc. in the at least one additional video frame that corresponds to at least one pixel, pixel group, etc. of the video frame mask (e.g., at least one pixel, at least one pixel group, etc. of the at least one additional video frame has the same position in the at least one additional video frame as the position of the at least one pixel, at least one pixel group, etc. of the video frame mask in the video frame mask). The controller 132 may cause a parameter value associated with the at least one pixel, at least one pixel group, etc. of the at least one additional video frame to change. For example, the controller 132 may cause a brightness value of the at least one pixel, at least one pixel group, etc. to have a particular value (e.g., a brightness value associated with a particular amount of brightness, such as an absence of brightness). As another example, the controller 132 may cause a color value of the at least one pixel, at least one pixel group, etc. to have a particular value (e.g., a color value associated with a particular color, such as black).

[0042] In additional examples, controller 132 may cause at least one region of at least one additional video frame corresponding to the video frame mask to have a particular color, such as black, and / or may blur one or more edges of at least one region (e.g., fade one or more edges from black to gray).

[0043] In some implementations, after applying the video frame mask to the at least one additional video frame, the controller 132 processes the at least one additional video frame using an object detection image processing technique (e.g., based on applying the video frame mask to the at least one additional video frame). For example, the controller 132 may process the at least one additional video frame using an object detection image processing technique (e.g., a Viola-Jones object detection technique, a scale-invariant feature transform technique, etc.) to detect an object (e.g., another work machine, another vehicle, a person, a building, etc.). The controller 132 may determine (e.g., based on information about the motion of the work machine 100, information about the motion of the at least one component, etc.) whether the object is in the motion path of the work machine 100 or the component (e.g., the implement 108).

[0044] Based on determining that the object is in the motion path of the work machine 100 or component, the controller 132 may issue an alert via the output device 128. For example, the controller 132 may send an alert signal or message to the output device 128, which causes the output device 128 to issue an audible alert via an audio output device of the output device 128, display a visual alert via a visual display of the output device 128, generate a vibration alert via a vibration output device of the output device 128, etc. The audible alert, visual alert, vibration alert, etc. may alert the operator of the work machine to change the motion path of the work machine 100 and / or component to avoid colliding with the object.

[0045] Additionally, or alternatively, the controller 132 may cause the movement of the work machine 100, the movement of the component, etc. to change based on determining that an object is in the path of movement of the work machine 100 or the component. For example, the controller 132 may send a control signal to the accelerator pedal 124 to change the amount of mechanical power delivered to the traction device 106, which may change the path of movement of the work machine and / or the traction device 106. As another example, the controller 132 may send a control signal to the implement control 126 to change the amount of pressure of the fluid supplied to and exhausted from the hydraulic actuators 120 and 122, which may change the path of movement of the implement 108. Thus, the controller 132 may send one or more control signals to other controllers, such as a machine direction controller, a deceleration pedal, etc., to change the path of the work machine 100 and / or the component of the work machine 100.

[0046] Although Figure 2 Example blocks of process 200 are shown, but in some implementations, process 200 may include more Figure 2 More blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in the process 200. Additionally, or alternatively, two or more blocks of process 200 may be performed in parallel.

[0047] As mentioned above, Figure 2is provided as an example. Other examples may differ from those in combination with Figure 2 as described.

[0048] Figure 3 is a diagram 300 of example video frames that may be captured by sensing device 130 and processed by controller 132. In these examples, sensing device 130 is attached to the rear of work machine 100 and captures video frames depicting implement 108 and the surrounding environment.

[0049] As shown in the example video frame 310, the work machine 100 may be moving in a forward direction (e.g., indicated by an arrow pointing in an upward direction associated with the surrounding environment), while the implement 108 may not be moving (e.g., indicated by a dot). In this case, the operator may engage the accelerator pedal 124 without engaging the implement control 126. In a manner similar to that described herein, the controller 132 may generate a composite video frame based on the example video frame 310 and other similar video frames corresponding to the duration. The controller 132 may determine at least one area of ​​the composite video frame associated with a lack of motion (e.g., at least one area associated with the dot) and generate a mask based on the at least one area.

[0050] As shown in the example video frame 320, the work machine 100 may be moving in a forward direction (e.g., indicated by an arrow pointing in an upward direction associated with the surrounding environment), while the implement 108 may be moving downward (e.g., indicated by an arrow pointing in a downward direction associated with the implement 108). In this case, the operator may simultaneously engage the accelerator pedal 124 and the implement control 126. In a manner similar to that described herein, the controller 132 may generate a composite video frame based on the example video frame 320 and other similar video frames corresponding to the duration. The controller 132 may determine at least one area of ​​the composite video frame that is associated with information about the movement of the implement 108 (e.g., at least one area associated with the downward pointing arrow) and / or at least one area of ​​the composite video frame that is not associated with information about the movement of the work machine 100 (e.g., at least one area not associated with the upward pointing arrow). The controller 132 may generate a mask based on the at least one area.

[0051] As shown in the example video frame 330, the work machine 100 may be moving in a forward direction (e.g., indicated by a smaller arrow pointing in an upward direction associated with the surrounding environment), while the implement 108 may be moving upward (e.g., indicated by a larger arrow pointing in an upward direction associated with the implement 108). In this case, the operator may engage the accelerator pedal 124 and the implement control 126 simultaneously. As shown, the larger arrow indicates a greater magnitude of apparent motion (e.g., associated with the implement 108) than the magnitude of apparent motion indicated by the smaller arrow (e.g., associated with the surrounding environment). In a manner similar to that described herein, the controller 132 may generate a composite video frame based on the example video frame 330 and other similar video frames corresponding to the duration. The controller 132 may determine at least one region of the composite video frame associated with information about the motion of the implement 108 (e.g., at least one region associated with the larger arrow) and / or at least one region of the composite video frame not associated with information about the motion of the work machine 100 (e.g., at least one region not associated with the smaller arrow). The controller 132 may generate a mask based on the at least one region.

[0052] As shown in the example video frame 340, the work machine 100 may be stationary (e.g., indicated by a dot associated with the surrounding environment) and the implement 108 may be moving downward (e.g., indicated by an arrow associated with the implement 108 pointing in a downward direction). In this case, the operator may engage the implement control 126 instead of the accelerator pedal 124. In a manner similar to that described herein, the controller 132 may generate a composite video frame based on the example video frame 340 and other similar video frames corresponding to the duration. The controller 132 may determine at least one area of ​​the composite video frame that is associated with information about the movement of the implement 108 (e.g., at least one area associated with the downward pointing arrow) and / or at least one area of ​​the composite video frame that is not associated with information about the movement of the work machine 100 (e.g., at least one area not associated with a dot). The controller 132 may generate a mask based on the at least one area.

[0053] Thus, with respect to example video frames 310 , 320 , 330 , and 340 , controller 132 may generate a mask that may be used to exclude implement 108 from additional video frames based on motion information regarding the motion of work machine 100 , information regarding the motion of implement 108 , and the like.

[0054] As mentioned above, Figure 3 is provided as an example. Other examples may differ from those in combination with Figure 3 as described.

[0055] Figure 44 is a diagram of example components of device 400. Device 400 may correspond to output device 128, sensing device 130, and / or controller 132. In some implementations, output device 128, sensing device 130, and / or controller may include one or more devices 400 and / or one or more components of device 400. Figure 4 As shown, the apparatus 400 may include a bus 410 , a processor 420 , a memory 430 , a storage component 440 , an input component 450 , an output component 460 , and a communication interface 470 .

[0056] The bus 410 includes components that allow communication between components of the device 400. The processor 420 is implemented in hardware, firmware, and / or a combination of hardware and software. The processor 420 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. The processor 420 includes one or more processors that can be programmed to perform functions. The memory 430 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 420.

[0057] The storage component 440 stores information and / or software related to the operation and use of the device 400. For example, the storage component 440 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or other types of non-transitory computer-readable media and corresponding drives.

[0058] Input components 450 include components that allow device 400 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Input components 450 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 460 include components that provide output information from device 400 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0059] The communication interface 470 includes transceiver-like components (e.g., transceivers and / or separate receivers and transmitters) that enable the device 400 to communicate with other components, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 470 can allow the device 400 to receive information from another device and / or provide information to another device. For example, the communication interface 470 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0060] The device 400 may perform one or more processes as described herein. The device 400 may perform these processes based on the processor 420 executing software instructions stored by a non-transitory computer-readable medium (e.g., memory 430 and / or storage component 440). A computer-readable medium is defined herein as a non-transitory memory device. The memory device includes memory space within a single physical memory device or memory space spread across multiple physical memory devices.

[0061] The software instructions may be read into the memory 430 and / or storage component 440 from another computer-readable medium or from another device via the communication interface 470. When executed, the software instructions stored in the memory 430 and / or storage component 440 may cause the processor 420 to perform one or more processes described herein. Hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0062] Figure 4 The number and arrangement of components shown are provided as examples. In practice, the apparatus 400 may include Figure 4 Additional components, fewer components, different components, or differently arranged components than those shown in . One set of components (eg, one or more components) of device 400 may perform one or more functions described as being performed by another set of components of device 400 .

[0063] Figure 5 is a flow chart of an example process 500 for excluding parts of a work machine from a video frame based on motion information. In some implementations, Figure 5 One or more process blocks of may be performed by a controller (eg, controller 132). In some implementations, Figure 5 One or more process blocks may be performed by another device or group of devices that is separate from or includes the controller, such as a sensing device (e.g., sensing device 130), an output device (e.g., output device 128), and the like.

[0064] like Figure 5As shown, process 500 may include obtaining video data including a plurality of video frames (block 510). For example, as described above, a controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may obtain video data including a plurality of video frames.

[0065] like Figure 5 As further shown in FIG. 5 , process 500 may include processing the plurality of video frames to determine apparent motion for each pixel of one or more pixels or for each pixel group of one or more pixel groups of each video frame of the plurality of video frames (block 520). For example, as described above, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may process the plurality of video frames to determine apparent motion for each pixel of one or more pixels or for each pixel group of one or more pixel groups of each video frame of the plurality of video frames.

[0066] like Figure 5 As further shown in FIG. 5 , process 500 may include selecting one or more processed video frames of the plurality of processed video frames that correspond to the duration (block 530). For example, a controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may select one or more processed video frames of the plurality of processed video frames that correspond to the duration, as described above.

[0067] like Figure 5 As further shown in FIG. 5 , process 500 may include generating a composite video frame based on the one or more processed video frames (block 540). For example, as described above, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may generate a composite video frame based on one or more processed video frames.

[0068] like Figure 5 As further shown in FIG. 5 , process 500 may include generating a video frame mask based on the composite video frame (block 550). For example, as described above, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may generate a video frame mask based on the composite video frame.

[0069] like Figure 5As further shown in FIG. 5 , process 500 may include obtaining additional video data including at least one additional video frame (block 560). For example, as described above, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may obtain additional video data including at least one additional video frame.

[0070] like Figure 5 As further shown in FIG. 5 , process 500 may include causing the video frame mask to be applied to at least one additional video frame (block 570). For example, as described above, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may cause the video frame mask to be applied to at least one additional video frame.

[0071] like Figure 5 As further shown in FIG. 5 , process 500 may include processing at least one additional video frame using an object detection image processing technique and based on applying a video frame mask to the at least one additional video frame (block 580). For example, the controller (e.g., using processor 420, memory 430, storage component 440, input component 450, output component 460, communication interface 470, etc.) may process at least one additional video frame using an object detection image processing technique and based on applying a video frame mask to the at least one additional video frame.

[0072] Process 500 may include additional implementations, such as any single implementation or any combination of implementations described in conjunction with one or more other processes described elsewhere herein.

[0073] Although Figure 5 Example blocks of process 500 are shown, but in some implementations, process 500 may include more Figure 5 More blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in the process 500. Additionally, or alternatively, two or more blocks of process 500 may be performed in parallel.

[0074] Industrial Applicability

[0075] A controller of a work machine may use object detection processing techniques to detect non-work machine objects in one or more images (e.g., a video frame stream associated with video data) and present an alert to an operator of the work machine based on the detection of the non-work machine object. However, in some cases, the one or more images may include one or more regions associated with a component of the work machine (e.g., an implement of the work machine). In these cases, the one or more regions may interfere with the ability of the controller to accurately detect the non-work machine object using the object detection processing techniques.

[0076] For example, the controller may mistakenly identify a component as a non-working machine object. This may result in a false positive alert being presented to the operator. As another example, the controller may fail to identify a non-working machine object that appears in close proximity to a component in one or more images. This may result in an alert not being presented to the operator when an alert should be presented. The presentation of a false positive alert and / or the failure to present a related alert to the operator may affect the operator's ability to effectively operate the working machine. This may cause the operator to operate the working machine in a manner that causes the working machine and / or the component to collide with a non-working machine object, which may damage the working machine, other working machines, other vehicles and / or other property.

[0077] Thus, some implementations described herein provide a controller that processes a plurality of video frames contained in video data to determine apparent motion of each pixel in one or more pixels or each pixel group in one or more pixel groups of each video frame in the plurality of video frames. The controller may select one or more processed video frames in the plurality of processed video frames corresponding to a duration, and may generate a composite video frame based on the one or more processed video frames. The controller may generate a video frame mask based on the composite video frame, and may obtain additional video data including at least one additional video frame. The controller may cause the video frame mask to be applied to at least one additional video frame, and may process the at least one additional video frame using an object detection image processing technique.

[0078] In this way, based on the motion information, the controller excludes the parts of the working machine from the video frame. This can improve the accuracy and / or reliability of the controller's detection of non-working machine objects using object detection image processing technology because the parts are excluded from the video frame. This can enable the controller to present relevant and / or timely alerts about non-working machine objects to the operator of the working machine, which can enable the operator to operate the working machine in a manner that enables the working machine and / or parts to avoid collision with non-working machine objects. Additionally, or alternatively, this can enable the controller to automatically control the motion path of the working machine and / or parts, which can enable the working machine and / or parts to avoid collision with non-working machine objects. This can reduce the possibility of damage to the working machine, other working machines, other vehicles, other property, etc.

Claims

1. A method comprising: Obtaining, by means (132), video data comprising a plurality of video frames (215); processing, by the device (132), the plurality of video frames (215) to determine a respective motion associated with each of the plurality of video frames; selecting, by the device (132), one or more processed video frames (225) from among the plurality of processed video frames corresponding to a period of time; generating, by the device (132), a composite video frame (235) indicating motion associated with the one or more processed video frames (225); Generating, by the device (132), a video frame mask (255) based on the composite video frame (235); obtaining, by the device (132), additional video data including at least one additional video frame (265); causing, by the device (132), the video frame mask (255) to be applied to the at least one additional video frame (265); and processing, by the device (132), the at least one additional video frame (265) using an object detection image processing technique and based on applying the video frame mask (255) to the at least one additional video frame (265), Wherein, generating the video frame mask (255) comprises: determining at least one area of ​​the composite video frame associated with a lack of motion; and The video frame mask is generated based on the at least one region (255).

2. The method of claim 1 , wherein processing the plurality of video frames (215) to determine a respective motion associated with each of the plurality of video frames comprises: The plurality of video frames are processed using an optical flow image processing technique (215).

3. The method of any one of claims 1 to 2, wherein generating the composite video frame (235) indicating motion associated with the one or more processed video frames (225) comprises: The composite video frame (235) is generated by determining an average motion associated with the one or more processed video frames (225).

4. The method according to any one of claims 1 to 3, wherein the composite video frame (235) is associated with a work machine (100), Generating the video frame mask (255) includes: Obtaining information about the movement of the work machine (100); as well as The video frame mask is generated based on the movement of the work machine (100).

5. An apparatus (132), comprising: one or more memories; and One or more processors, communicatively coupled to the one or more memories, to: Obtaining video data including a plurality of video frames (215); processing the plurality of video frames (215) to determine a respective motion associated with each video frame of the plurality of video frames; selecting one or more processed video frames of the plurality of processed video frames corresponding to a period of time (225); generating a composite video frame (235) indicating motion associated with the one or more processed video frames (225); Determining at least one region of a composite video frame associated with at least one component (106, 108, 112) of a work machine (100); generating a video frame mask (255) based on the composite video frame (235) and the at least one region of the composite video frame (235); obtaining additional video data including at least one additional video frame (265); causing the video frame mask (255) to be applied to the at least one additional video frame (265); and processing the at least one additional video frame (265) using an object detection image processing technique and based on applying the video frame mask (255) to the at least one additional video frame (265), in, Generating the video frame mask (255) comprises: determining at least one area of ​​the composite video frame associated with a lack of motion; and The video frame mask is generated based on the at least one region (255).

6. The apparatus (132) of claim 5, wherein when causing the video frame mask (255) to apply to the at least one additional video frame (265), the one or more processors are configured to: For each pixel of a plurality of pixels included in the at least one additional video frame (265), a parameter associated with the pixel is caused to have a particular value.

7. The apparatus (132) of any one of claims 5 to 6, wherein when determining the at least one region of the composite video frame (235), the one or more processors are configured to: obtaining information about movement of at least one component (106, 108, 112) of the work machine (100); and The at least one region of the composite video frame (235) is determined based on information regarding the motion of the at least one component (106, 108, 112).

8. The apparatus (132) of any one of claims 5 to 7, wherein when determining the at least one region of the composite video frame (235), the one or more processors are configured to: Obtaining one or more or control instructions regarding the movement of a machine (108) or work machine (100); determining a magnitude and direction of movement associated with the implement (108) based on the one or more control instructions; and The at least one region of the composite video frame (235) corresponding to the magnitude and direction of motion associated with the implement (108) is determined.

9. The apparatus (132) of any one of claims 5 to 8, wherein when processing the at least one additional video frame (265) using the object detection image processing technique, the one or more processors are configured to: processing the at least one additional video frame (265) using the object detection image processing technique to detect an object; determining whether the object is in a path of motion of the work machine (100) or implement (108); and Based on determining that the object is in the motion path of the work machine (100) or implement (108), an alert is displayed on a display of the work machine (100).

Images