Image display method for self-propelled device and self-propelled device

Abstract

An image display technology and application for a self-propelled device, the self-propelled device includes an image acquisition component for acquiring images; a remote control terminal connected to the self-propelled device via wireless communication and capable of manipulating the self-propelled device; the remote control terminal includes a display component for displaying the image acquired by the image acquisition component; the image displayed by the display component includes a reference mark for indicating the relative position of the machine relative to the environment in the image displayed by the display component. The present invention provides an image display technology and an associated control method, which can accurately reflect the position of a lawn mower in a working area, facilitate users to clearly understand the real-time working conditions of the lawn mower robot, and guide users to control the machine, which can effectively improve the operability and safety of users controlling the machine through the remote control terminal.

Description

Technical Field

[0001] The invention relates to the field of garden tools, and in particular to an image display method for a self-propelled device and the self-propelled device. Background Art

[0002] A lawn mower robot is a common automatic walking device that can move autonomously outdoors continuously and in real time. As a type of automatic walking device, the intelligent lawn mower robot is suitable for cleaning vegetation on hills, terraces, plains and other plots, as well as weeds in lawns. It has the advantages of simple operation and high work efficiency. Our common schools and street green space beautification are all completed with these automatic walking devices. In foreign countries, many families use automatic walking devices to weed their gardens.

[0003] When the lawn mower robot works for the first time, it needs to walk along the boundary of the working area to collect the boundary data of the working area. In the initial working stage, the user is usually required to follow the lawn mower robot and control it to walk along the boundary of the working area by remote control. With the advancement of battery technology, the battery capacity of the lawn mower robot has increased, and the area in which the lawn mower robot can work is getting larger and larger. It takes a long time for the user to follow the lawn mower robot, and he also needs to walk a long distance. In order to solve this problem, some lawn mower robots use cameras to record the front image in real time and transmit it to the terminal in the user's hand or other remote control devices or apps. The user uses the real-time image to understand the operating status and real-time position of the lawn mower robot and complete the work of controlling the lawn mower robot to walk.

[0004] The images provided by the existing image transmission technology for users to refer to cannot accurately reflect the real-time position of the mowing robot, which is not conducive to user operation and has certain risks. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an image display method for a self-propelled device and a self-propelled device, which can accurately reflect the position of the lawn mower in the working area, so that the user can clearly understand the real-time working status of the lawn mower robot.

[0006] First, in order to achieve the above-mentioned purpose, an image display method for a self-propelled device is proposed, wherein the self-propelled device comprises

[0007] An image acquisition component, used for acquiring images;

[0008] A remote control terminal, which is connected to the self-propelled device through wireless communication and can control the self-propelled device;

[0009] The remote control terminal comprises a display component for displaying the image acquired by the image acquisition component;

[0010] The image displayed by the display component includes

[0011] A reference mark for indicating the relative position of the machine with respect to the environment in the image displayed by the display component;

[0012] The reference mark is a virtual image.

[0013] As a further improvement of this solution, the image displayed by the display component includes:

[0014] The real-time image area is used to display the real-time image collected by the image acquisition component;

[0015] A virtual image area, used to display a virtual supplementary image generated by real-time image parameters or a time-lapse image generated by historical images;

[0016] The historical image is acquired from the image data collected by the image acquisition component.

[0017] As a further improvement of the present solution, the virtual image area at least includes an area extending outward from the lower edge of the real-time image area.

[0018] As a further improvement of the present solution, the virtual supplementary image is generated by using pixel information at the edge of the real-time image or within a certain area at the edge.

[0019] As a further improvement of the present solution, the image acquisition component attaches timestamp data to each frame of image acquired, and selects a historical image containing timestamp information before a certain time interval as a time-delay image based on the timestamp information of the real-time image displayed in the real-time image area.

[0020] As a further improvement of this solution, the time-lapse image is obtained by the following method:

[0021] S1. The machine is started, the image acquisition unit is started and starts to record images, and the timestamp and distance data are recorded on each frame of the image. The distance data of the first frame of the image acquired after the image acquisition unit is started is 0, and the distance data of each frame of the image is the accumulated actual driving distance data since the machine was started;

[0022] S2. Obtain the distance data recorded in the real-time image, and fill the virtual image area with the corresponding historical image. The distance data in the corresponding historical image is the distance data in the real-time image minus the specific length in the environment corresponding to the image displayed in the virtual image area.

[0023] As a further improvement of the present solution, the time-lapse image is an image captured from the bottom edge of the historical image and having the same pixel height and width as the virtual image area.

[0024] As a further improvement of the present solution, when the machine starts to run, the virtual image area is filled with a virtual supplementary image, and when there is a historical image that meets the use conditions, the virtual image area is filled with an extended image.

[0025] As a further improvement of the present solution, the virtual image area below the real-time image area is filled with a delayed image, and areas at other positions of the virtual image area are filled with a virtual supplementary image.

[0026] As a further improvement of the present solution, at the boundary between the virtual image area and the real-time image area, the image in the real-time image area and / or the virtual image area may be blurred, softened or otherwise processed.

[0027] A self-propelled device, using the image display method, the remote control terminal comprises:

[0028] The user situation collection component may be a camera or other components, and is used to collect user data, including face data, eyeball data, and pupil data.

[0029] As a further improvement of the present solution, when the camera element cannot collect facial data or collects facial data but cannot collect eyeball data / pupil data, the operating state of the machine cannot be changed through the remote control terminal, and if the operating component in the remote control terminal is touched at this time, the machine enters a stop state.

[0030] As a further improvement of the present solution, when a remote control terminal is used to control the machine, the remote control terminal compares the time difference between the timestamp data of the real-time image displayed by the display component and the internal clock / network clock of the remote control terminal. When the time difference exceeds the preset time, or the timestamp data of the real-time image does not change for more than the specified time, the machine is stopped and the machine operating status cannot be changed through the remote control terminal. Beneficial Effects

[0031] The present invention provides an image display method, which can accurately reflect the position of a lawn mower in a working area, so that a user can clearly understand the real-time working situation of the lawn mowing robot and guide the user to control the machine.

[0032] By utilizing the image display method of the present invention, the operability and safety of a user controlling a machine through a remote control terminal can be effectively improved.

[0033] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0035] Figure 1 is the Figure 1 Schematic diagram of the control terminal.

[0036] Figure 2 It is a schematic diagram of the image acquisition unit acquiring an image.

[0037] Figure 3 is a schematic diagram of the reference mark.

[0038] Figure 4 It is a schematic diagram of the actual position of the machine.

[0039] Figure 5 The real-time image area and the virtual image area are schematically shown in FIG. Figure 1 .

[0040] Figure 6 The real-time image area and the virtual image area are schematically shown in FIG. Figure 2 .

[0041] Figure 7 The real-time image area and the virtual image area are schematically shown in FIG. Figure 3 .

[0042] Figure 8 It is a schematic diagram of the remote control terminal with the user situation collection component.

[0043] The accompanying drawings are marked as follows:

[0044] 100. Machine;

[0045] 200, image acquisition component;

[0046] 300, remote control terminal; 301, operating component;

[0047] 400, display component; 401, real-time image area; 402, virtual image area;

[0048] 500, reference mark; 500', actual position of the machine;

[0049] 700. User situation collection component. DETAILED DESCRIPTION

[0050] In order to make the purpose and technical solution of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0052] The meaning of "and / or" in the present invention means that both the cases where each exists alone or both exist at the same time are included.

[0053] The meaning of "inside" and "outside" in the present invention refers to the direction from the signal line to the inside of the working area is inside, and the opposite direction is outside, relative to the signal line itself; it is not a specific limitation on the device mechanism of the present invention.

[0054] The term “connection” as used in the present invention may mean a direct connection between components or an indirect connection between components via other components.

[0055] like Figure 1 As shown, in the existing remote control device, the remote control terminal 300 is provided with a manipulation component 301 and a display component 400 , and the image displayed by the display component 400 is the image / video etc. acquired by the image acquisition component 200 on the machine 100 .

[0056] like Figure 2 As shown, the image acquisition component 200 is usually set on the machine 100. Due to the viewing angle of the image acquisition component 200, the body of the machine 100 is usually in a blind spot in the image acquired by the image. The image seen by the user through the display component 400 usually does not include the body of the machine 100. When the user operates with the image displayed by the display component 400, the lack of reference to the body may lead to problems such as inaccurate operation.

[0057] In order to solve the above problems, Figure 3As shown, a reference mark 500 is added to the lower part of the image displayed by the display component 400 to provide the user with a reference for the position of the machine 100 relative to the environment displayed in the image. Based on the reference mark 500, the user can intuitively and accurately understand the position of the machine 100 in the environment displayed in the image, which provides a reference for subsequent control.

[0058] However, if Figure 2 As shown, the image captured by the image acquisition component 200 on the machine 100 is usually a real-time image within the imaging area, and the display component 400 displays the real-time image to the user. Due to the existence of the blind spot, compared with the environment displayed by the real-time image, the actual position of the machine 100 should be outside the environment displayed by the image. Specifically, the actual position of the machine 100 should be below the environment displayed by the image, such as Figure 4 Therefore, if only the reference mark 500 is added to the lower part of the image displayed by the display component 400, and the user uses the reference mark 500 as a reference, a certain position error may occur.

[0059] In order to solve the above problems, the following image display technology is provided.

[0060] like Figure 5 As shown, the display area of ​​the display component 400 includes a real-time image area 401 and a virtual image area 402. The virtual image area 402 is an area extending outward from the edge of the real-time image area 401. The virtual image area 402 at least includes an area extending outward from the lower edge of the real-time image area 401. Figure 6 and 7 The figure shows the situation that the virtual image area 402 extends outward from the edges around the real-time image area 401 and the situation that the virtual image area 402 extends outward from the bottom and side edges of the real-time image area 401.

[0061] The real-time image area 401 displays the real-time image collected by the image acquisition component 200200, the image displayed in the virtual image area 402 is a virtual supplementary image or a delayed image, and the reference mark 500 is located in the virtual image area 402. Based on the above image display technology, the specific position of the machine 100 relative to the environment displayed by the real-time image can be displayed more accurately and objectively.

[0062] For the virtual image area 402, when the image displayed therein is a virtual supplementary image, the virtual supplementary image is generated using the real-time image data in the real-time image area 401. Specifically, the virtual supplementary image is generated using the pixel information at the edge of the real-time image or within a certain area at the edge. Figure 5As shown, the virtual image area 402 is located below the real-time image area 401, and the image in the virtual image area 402 is generated by the pixel information in the lower edge of the real-time image area 401 or the area of ​​several pixels of the lower edge, and specifically, the image is generated by using the color information of the pixels. Since the lawn mower 100 usually travels in a grassland environment, the lower part of the real-time image area 401 is mainly grassland, and the image information below the real-time image area 401 is used in the virtual image area 402 to simulate the grassland image and merge it with the real-time image area 401.

[0063] In another implementation, the image in the virtual image area 402 is filled with the historical image collected by the image acquisition component 200. Generally, the image acquisition component 200 will add timestamp data to each frame of the image acquired, select the historical image containing the timestamp information before a certain time interval according to the timestamp information of the real-time image displayed in the real-time image area 401, and fill part of the image information in the historical image into the virtual image area 402 for display. The image of the historical image, which is cut from the bottom edge and has the same pixel height and width as the virtual image area 402, is filled into the virtual image area 402.

[0064] The specific time interval may be a fixed time interval set manually, such as a time interval determined during the design phase of the machine 100 and stored in the main control program of the machine 100. This method may be used for some lawn mowing machines 100 with a fixed driving speed.

[0065] Alternatively, the time interval can be obtained by comprehensive calculation based on the driving speed of the machine 100 and the specific length of the environment corresponding to the image displayed in the virtual image area 402. Specifically, the length of the virtual image area 402 is fixed, and the actual length of the environment image displayed in the virtual image area 402 in the actual environment can be obtained through experiments, calculations, etc., which may be related to the camera viewing angle and placement position. After obtaining the actual length of the environment image displayed in the virtual image area 402 in the actual environment, the corresponding time interval is calculated according to the actual driving speed of the machine 100. If the driving speed is variable, the time interval can be obtained by segmented calculation or by the following method:

[0066] 1. The machine 100 is started, the image acquisition unit is started and starts to record images, and the timestamp and distance data are recorded on each frame of the image. The distance data of the first frame of the image acquired after the image acquisition unit is started is 0, and the distance data of each frame of the image is the accumulated actual driving distance data since the machine 100 is started;

[0067] 2. Obtain the distance data recorded in the real-time image, and fill the virtual image area 402 with the corresponding historical image. The distance data in the corresponding historical image is the distance data in the real-time image minus the specific length in the environment corresponding to the image displayed in the virtual image area 402.

[0068] The accumulated actual driving distance data can be calculated based on the number of motor rotations and the diameter of the driving wheel.

[0069] In the above two implementations, at the junction of the virtual image area 402 and the real-time image area 401, the image in the real-time image area 401 and / or the virtual image area 402 can be blurred, softened, etc., so that the transition from the real-time image area 401 to the virtual image area 402 is relatively smooth. When the user watches the image displayed by the display component 400, there will be no obvious difference between the real-time image area 401 and the virtual image area 402, resulting in a poor visual effect.

[0070] like Figure 6 and 7 In the situation shown, the virtual image area 402 includes extensions of multiple edge areas of the real-time image area 401. In actual use, except that the virtual image area 402 below the real-time image area 401 can be filled with historical image data, areas at other positions of the virtual image area 402 need to be filled with virtual supplementary images.

[0071] Since there is not enough historical image data to provide for the virtual image area 402 to use when the machine 100 is just started, virtual supplementary images are used to fill the area when the machine 100 starts. When historical images that meet the usage conditions are available, the virtual image area 402 is filled with historical images.

[0072] Since the image acquisition unit may be capable of positional changes, such as rotation, up and down swinging, etc., when the position of the image acquisition unit changes, the specific length of the environment displayed in the virtual image area 402 changes accordingly, and the specific position of the actual position of the machine 100 relative to the lower edge of the real-time image changes, and the above changes will be reflected in real time in the imaging of the virtual image area 402. The specific distance of the actual position of the machine 100 relative to the lower edge of the real-time image can be obtained through the viewing angle, installation position and installation angle of the image acquisition unit.

[0073] like Figure 8As shown, on the display component 400, other virtual signs showing the working conditions of the machine 100 are also provided, including a speed sign 403 showing the running speed of the machine 100, a heading sign 404 showing the turning direction of the machine 100, a mowing speed sign 405 showing the speed of the mowing disc of the machine 100, and a power sign 408 showing the battery power of the machine 100. The running speed of the machine 100 can be obtained by the speed and wheel diameter of the driving motor of the machine 100, the turning direction of the machine 100 can be obtained by the speed difference between the two driving motors of the machine 100, and the speed of the mowing disc of the machine 100 can be obtained by the speed of the mowing motor of the machine 100.

[0074] The remote control terminal 300 is also provided with a user situation collection component, which can be a camera or other components, used to collect user data, including face data, eyeball data, pupil data, etc. When the camera component cannot collect face data or collects face data but cannot collect eyeball data / pupil data, the operation state of the machine 100 cannot be changed through the remote control terminal 300, and if the operating component 301 in the remote control terminal 300 is touched at this time, the machine 100 enters a stop state.

[0075] When the user uses the remote control terminal 300 to control the machine 100, the remote control terminal 300 compares the time difference between the timestamp data of the real-time image displayed by the display component 400 and the internal clock / network clock of the remote control terminal 300. When the time difference exceeds the preset time, or the timestamp data of the real-time image does not change for more than the specified time, the machine 100 is stopped and the operating status of the machine 100 cannot be changed through the remote control terminal 300.

[0076] The above two methods are both intended to prevent the user from operating the machine 100 through the remote control terminal 300 when not observing the display component 400, thereby reducing the possibility of danger. At the same time, it can also prevent the user from accidentally touching the remote control terminal 300, resulting in erroneous operation of the machine 100.

[0077] The above is only an embodiment of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. An image display method for a self-propelled device, wherein the self-propelled device comprises an image acquisition component for acquiring images; a remote control terminal is connected to the self-propelled device via a wireless communication method and can operate the self-propelled device; the remote control terminal comprises a display component for displaying the image acquired by the image acquisition component; characterized in that: The image displayed by the display component includes a reference mark for indicating the relative position of the machine relative to the environment in the image displayed by the display component; The reference marker is a virtual image, and the image displayed by the display component includes: a real-time image area, which is used to display the real-time image collected by the image acquisition component; a virtual image area, which is used to display a virtual supplementary image generated by real-time image parameters or a delayed image generated by a historical image; the historical image is acquired from the image data collected by the image acquisition component; the virtual image area at least includes an area extending outward from the lower edge of the real-time image area; the virtual supplementary image is generated using pixel information at the edge of the real-time image or within a certain area at the edge; the image acquisition component attaches timestamp data to each frame of the image acquired, and selects a historical image containing timestamp information before a certain time interval as a delayed image based on the timestamp information of the real-time image displayed in the real-time image area.

2. The image display method for a self-propelled device according to claim 1, characterized in that: The time-lapse image is obtained by the following method: S1, the machine is started, the image acquisition unit is started and starts to record images, and the timestamp and distance data are recorded on each frame of the image. The distance data of the first frame of the image acquired after the image acquisition unit is started is 0, and the distance data of each frame of the image is the accumulated actual driving distance data from the start of the machine; S2. Obtain the distance data recorded in the real-time image, and fill the virtual image area with the corresponding historical image. The distance data in the corresponding historical image is the distance data in the real-time image minus the specific length in the environment corresponding to the image displayed in the virtual image area.

3. The image display method for a self-propelled device according to claim 2, characterized in that: The time-lapse image is an image captured from the lower edge of the historical image and having the same pixel height and width as the virtual image area.

4. The image display method for a self-propelled device according to claim 3, characterized in that: When the machine starts to run, the virtual image area is filled with a virtual supplementary image. When there is a historical image that meets the use conditions, the virtual image area is filled with an extended image.

5. The image display method for a self-propelled device according to claim 4, characterized in that: The virtual image area below the real-time image area is filled with the delayed image, and the area at other positions of the virtual image area is filled with the virtual supplementary image.

6. The image display method for a self-propelled device according to any one of claims 2 to 5, characterized in that: At the junction of the virtual image area and the real-time image area, blurring and softening operations are performed on the images in the real-time image area and / or the virtual image area.

7. A self-propelled device, characterized in that: Using the image display method as claimed in any one of claims 1 to 6, the remote control terminal comprises: a user situation collection component, the user situation collection component is a camera, used to collect user data, the user data includes face data, eyeball data, pupil data.

8. A self-propelled device as claimed in claim 7, characterized in that: When the camera component cannot collect facial data or collects facial data but cannot collect eye data / pupil data, the machine's operating state cannot be changed through the remote control terminal, and if the operating component in the remote control terminal is touched at this time, the machine enters a stop state.

9. A self-propelled device as claimed in claim 8, characterized in that: When using a remote control terminal to control the machine, the remote control terminal compares the time difference between the timestamp data of the real-time image displayed by the display component and the internal clock / network clock of the remote control terminal. When the time difference exceeds the preset time, or the timestamp data of the real-time image does not change for more than the specified time, the machine is stopped and the machine operating status cannot be changed through the remote control terminal.

Images