System and method for providing augmented reality navigation screen for agricultural vehicle

Abstract

Embodiments relate to a system and a method for providing an augmented reality (AR) navigation screen provided in an agricultural vehicle, which comprise: acquiring GPS location information of the agricultural vehicle through a GPS module; acquiring navigation information of the agricultural vehicle; acquiring a travel environment image acquired by capturing, through a camera module, the travel environment in which the agricultural vehicle travels; calculating travel-related information of the agricultural vehicle on the basis of the GPS location information of the agricultural vehicle acquired during traveling; acquiring, as travel error information of the agricultural vehicle, the difference between a travel guide path and an actual travel path of the agricultural vehicle; and generating at least one graphic object that visually expresses at least one from among the navigation information, the travel-related information and travel error information of the agricultural vehicle, so as to generate an AR navigation screen of the agricultural vehicle.

Description

System and method for providing an augmented reality navigation screen for agricultural vehicles

[0001] Embodiments of the present application relate to a technology for providing an augmented reality (AR) navigation screen for an agricultural vehicle.

[0002] With the advancement of autonomous driving and wireless communication technologies, there is an increasing trend of attaching autonomous driving devices to agricultural vehicles and utilizing GPS data and other sensing data from the autonomous driving devices to enable autonomous driving of agricultural vehicles.

[0003] However, because autonomous driving technology has not yet reached the level of fully autonomous driving, workers are required to ride in or walk alongside autonomous agricultural vehicles. It is necessary to provide driving-related information to workers using autonomous agricultural vehicles.

[0004] Recent advancements in augmented reality (AR) technology have led to a growing adoption of head-up displays (HUDs) in the automotive industry, which project driving routes onto a real-world background to provide navigation information. AR and VR technologies, which bridge the gap between reality and virtual reality, serve as mediators for providing more convenient and accurate guidance to users in the navigation field. While technically distinct, these two technologies are accelerating the advancement of navigation technology and contributing to an enhanced user experience. Augmented reality (AR) technology enhances the real-world experience by combining virtual elements with the real world. This technology uses cameras and sensors to detect the surrounding environment and, based on this, provides virtual graphics, audio, and information in real time. Users can view the real world alongside the virtual elements, allowing them to interact and gain information. AR is primarily used in navigation, gaming, education, healthcare, and marketing, and is implemented in various forms, such as through AR glasses and smartphone apps. AR technology merges the real and virtual worlds to provide a realistic experience, and advancements in computer vision and sensing technologies are enabling more realistic and diverse functions.

[0005] However, compared to the automobile industry, tractor manufacturers have been slow to adopt AR technologies such as HUDs, and currently mainly use virtual graphics or satellite maps to provide routes.

[0006] Meanwhile, for conventional passenger agricultural vehicles or autonomous agricultural vehicles, technology for accurately determining their location (positioning) is essential. The most widely used positioning system is the GNSS. GNSS (Global Navigation Satellite System) is a general term for systems that use satellites to provide information on the location, altitude, and velocity of objects on the ground. The United States' Global Positioning System (GPS) is a prime example.

[0007] In a typical environment where GPS is used, it would be possible to estimate the exact location and movement path of a vehicle using a big data-based map matching algorithm. However, in an environment where agricultural vehicles operate, it is difficult to apply such a map matching algorithm.

[0008] As a result, there is a problem that sometimes it is difficult to recognize the correct route due to a discrepancy between the navigation route and the actual reality.

[0009] Therefore, in an autonomous driving environment for agricultural vehicles, UX / UI technology is required to efficiently provide the operator with driving-related information regarding the error between the GPS point of the autonomous driving agricultural vehicle and the location on the map of the autonomous driving agricultural vehicle, and further, to accurately match the GPS point of the autonomous driving agricultural vehicle and the location on the map of the autonomous driving agricultural vehicle to enable efficient autonomous driving.

[0010] Based on the discussion described above, embodiments of the present application aim to provide a system and method for providing an AR navigation screen for an agricultural vehicle to efficiently provide workers with various driving-related information, including the error between the GPS point of the agricultural vehicle and the location of the agricultural vehicle on a map.

[0011] In addition, embodiments of the present application aim to provide a system and method capable of accurately matching a GPS point of an autonomous agricultural vehicle with a location on a map of the autonomous agricultural vehicle by calibrating an error between the GPS point and the location on a map of the autonomous agricultural vehicle in the driving-related information of the displayed agricultural vehicle.

[0012] A method for providing an augmented reality (AR) navigation screen according to one aspect of the present application can be performed by a system for providing an augmented reality (AR) navigation screen installed in an agricultural vehicle. The method comprises the steps of: obtaining GPS location information indicating a GPS location of the agricultural vehicle from a GPS module; obtaining navigation information of the agricultural vehicle, wherein the navigation information includes driving guidance path information describing a driving guidance path along which the agricultural vehicle will drive; obtaining a driving environment image photographing a driving environment in which the agricultural vehicle drives from a camera module; calculating driving-related information of the agricultural vehicle based on the GPS location information of the agricultural vehicle acquired while the agricultural vehicle is driving; obtaining a difference between the driving guidance path and the actual driving path of the agricultural vehicle as driving error information of the agricultural vehicle; The method may include a step of generating at least one graphic object that visually expresses at least one of the above navigation information, driving-related information of the agricultural vehicle, and driving error information, and generating an AR navigation screen in which the AR graphic object is placed on the driving environment image.

[0013] In one embodiment, the step of generating the AR navigation screen may include the steps of: identifying a position of a graphic object representing at least two pieces of information from among the navigation information, driving-related information, and driving error information, wherein the position of the AR graphic object is a position on the driving environment image; defining a shape space of each graphic object based on the identified position, and generating a shape of the AR graphic object based on the defined shape space; and overlaying the shape of the AR graphic object with the identified position on the driving environment image, thereby generating an AR navigation screen in which the shape of the AR graphic object is expressed on the driving environment image.

[0014] In one embodiment, the AR graphic object may include a first graphic object visually representing a driving guidance path. The step of identifying a position of the AR graphic object on the driving environment image may include a step of identifying a position of the first graphic object based on GPS position information of the agricultural vehicle, map information for a geographical area including the GPS position of the agricultural vehicle, driving direction information of the agricultural vehicle, and parameters of the camera module. Here, the step of identifying the position of the first graphic object may include a step of mapping a two-dimensional coordinate system for the driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world; a step of calculating a three-dimensional position of the driving guidance path in the virtual three-dimensional world based on the map information and the driving guidance path information; and a step of changing the three-dimensional position of the driving guidance path into a two-dimensional position of the driving guidance path through a mapping relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, and identifying the changed two-dimensional position as the position of the first graphic object.

[0015] In one embodiment, the step of mapping a two-dimensional coordinate system in the driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world may include a step of mapping the two-dimensional coordinates of the driving environment image and the virtual three-dimensional world coordinates by setting internal parameters of the camera module. The intrinsic parameters of the camera module may include one or more of a principal point, a focal length, and a distortion coefficient.

[0016] In one embodiment, the driving-related information includes at least one of driving mode information, a driving route driven along the driving guidance route, a driving distance along the driving route, a driving time, real-time driving speed of the agricultural vehicle at the current location at the time of generating the AR navigation screen, a driving direction angle of the agricultural vehicle, and work result area information. The AR graphic object further includes at least one graphic object of a second graphic object visually representing a work result area according to the driving of the agricultural vehicle, a third graphic object visually representing a driving direction, and a fourth graphic object visually representing the driving error information. The step of identifying the position of the AR graphic object on the driving environment image includes, when the AR graphic object includes the second graphic object, a step of identifying the position of the second graphic object based on the position of the identified first graphic object, driving route information, driving distance information, and pre-stored unit work result area information per unit driving distance; If the AR graphic object includes the third graphic object, the step of identifying the position of the third graphic object based on the position and driving direction information of the identified first graphic object may be included; and if the AR graphic object includes the fourth graphic object, the step of identifying the position of the fourth graphic object based on the position of the identified first graphic object and the driving error information may be included. The positions of the second graphic object, the third graphic object, and the fourth graphic object are positions on a two-dimensional coordinate system with respect to the driving environment image.

[0017] In one embodiment, the step of generating the AR navigation screen may further include the steps of: generating a map view area showing GPS location information of the agricultural vehicle, a map image, driving direction information of the agricultural vehicle, work result area information, and a surrounding area of ​​the agricultural vehicle; generating an error view area describing a current driving error of the agricultural machine at the time of generating the AR navigation screen based on the driving error information; and overlaying at least one of the map view area and the error view area on the driving environment image to generate the AR navigation screen showing the map view area and the error view area.

[0018] In one embodiment, the step of generating the AR navigation screen may further include the step of displaying driving distance information of the agricultural vehicle, the area of ​​the work result area, and driving time information in the generated driving-related information in a first sub-area of ​​the navigation screen; the step of displaying driving mode information of the agricultural vehicle in a second sub-area of ​​the navigation screen; and the step of displaying the content of a driving guidance message in the acquired navigation information in a sub-area of ​​the navigation screen.

[0019] In one embodiment, the method may further include a step of displaying an AB driving reference line on a calibration interface screen including a map image on which an agricultural vehicle appears based on a GPS location of the agricultural vehicle, in order to eliminate an over-the-earth error (OTE) of the agricultural vehicle, and calibrating the GPS coordinates of the agricultural vehicle to actual coordinates of the agricultural vehicle using the AB driving reference line. The AB driving reference line is at least a portion of a driving guidance path in the navigation information. In the AR navigation screen, in order to display the driving error of the agricultural vehicle corrected according to the calibration result, an AR graphic object indicating the driving error of the agricultural vehicle corrected according to the calibration result on the AR navigation screen is relatively reduced compared to before calibration.

[0020] In one embodiment, the calibrating step may include: displaying a baseline movement icon on the calibration interface screen so as to overlap the AB driving baseline; moving the baseline movement icon displayed on the calibration interface screen in response to a user input to the calibration interface screen; and, when the baseline movement icon displayed on the calibration interface screen is moved in response to the user input to the calibration interface screen within a preset movement section, additionally displaying an AB driving baseline movement indicator on the calibration interface screen. The AB driving baseline movement indicator is located in an area outside the left area and the right area on the calibration interface screen, and when the movement of the baseline movement icon is performed outside the preset movement section, the display of the AB driving baseline movement indicator is deactivated. The user input for the calibration interface screen is a user's touch input for a preset area based on a touch interface built into the calibration interface screen, and a left area or a right area based on the AB driving reference line on the calibration interface screen is an area set to receive the user's touch input, and when the user's touch input is made in the left area, the reference line movement icon is moved to the left by a preset distance, and when the user's touch input is made in the right area, the reference line movement icon is moved to the right by a preset distance. The AB driving reference line movement indicator includes an auxiliary AB driving reference line and an auxiliary moving line icon that serve as a reference for movement of an auxiliary moving line icon, and the auxiliary AB driving reference line and the auxiliary moving line icon are displayed together on the AB driving reference line and the calibration interface screen.The above auxiliary AB driving reference line is displayed in the center of the AB driving reference line movement indicator, and the auxiliary moving line icon can move in response to a touch input to the left area or the right area of ​​the AB driving reference line movement indicator that causes movement of the reference line movement icon.

[0021] In one embodiment, in the step of displaying the AB driving reference line (720) on the map (710) based on the GPS coordinates on the calibration interface screen provided by the second display or the user terminal, when a calibration button (713) that triggers calibration displayed on the map (710) is input, a modal (730) for calibration is displayed by overlapping it on the map (710), and the modal (730) for calibration displays the AB driving reference line (720), the reference line movement icon (740), and the AB driving reference line movement indicator (750), and the left area (731) and the right area (732) can be located on the modal (730).

[0022] According to another aspect of the present application, a computer-readable recording medium can record a program for performing a method for generating an augmented reality (AR) navigation screen according to the above-described embodiments.

[0023] According to another aspect of the present application, a navigation module for generating an augmented reality (AR) navigation screen is installed in an agricultural vehicle. The navigation module stores a processor and a memory including instructions, which instructions, when executed by the processor, perform the following steps: obtaining GPS location information indicating the GPS location of the agricultural vehicle from a GPS module; obtaining navigation information of the agricultural vehicle, wherein the navigation information includes driving guidance route information describing a driving guidance route along which the agricultural vehicle will drive; obtaining a driving environment image captured from a camera module of a driving environment in which the agricultural vehicle drives; calculating driving-related information of the agricultural vehicle based on GPS location information of the agricultural vehicle acquired while the agricultural vehicle is driving; obtaining a difference between the driving guidance route and the actual driving route of the agricultural vehicle as driving error information of the agricultural vehicle; The method may include a command to perform a step of generating at least one graphic object that visually expresses at least one of the above navigation information, driving-related information of the agricultural vehicle, and driving error information, and generating an AR navigation screen in which the AR graphic object is placed on the driving environment image.

[0024] A system for displaying driving-related information of an agricultural vehicle according to various embodiments of the present invention uses augmented reality technology to create various driving-related information, including an error between a GPS point of an autonomous agricultural vehicle and a location on a map of the autonomous agricultural vehicle, as an augmented reality object in an autonomous driving environment of the agricultural vehicle, and outputs an interface screen for augmented reality calibration in which the augmented reality object is placed on a driving image, thereby efficiently providing various driving-related information to a worker.

[0025] Specifically, agricultural vehicle drivers can view accurate driving routes that are consistent with reality through real-time front and rear display screens utilizing AR technology. This allows users to receive more intuitive and accurate guidance than existing navigation systems, which often have a large visual error range compared to reality, and can enhance tractor autonomous driving and work efficiency. The navigation screen uses AR technology to combine real-time video and driving information, allowing users to view accurate driving routes without any sense of disconnect. Users can view essential information, including the driving route, through the display screen. By overlaying the route with graphics onto the actual environment, the AR navigation screen provides users with accurate guidance, facilitating effective operation. As a result, utilizing the AR navigation screen realistically displays the real-time driving route and related information of the tractor performing agricultural work, allowing users to intuitively understand the information. This has the effect of enhancing the usability of tractor autonomous driving and agricultural work.

[0026] Additionally, the system displaying driving-related information for the aforementioned agricultural vehicle can use the AB driving reference line to match GPS points with the map location of the autonomous agricultural vehicle. This process allows users to clearly and easily move the AB driving reference line to the desired location, even upon first use, thereby enhancing app usability.

[0027] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present application belongs from the description below.

[0028] FIG. 1 is a network environment diagram of a system (10) that provides an AR navigation screen for an agricultural vehicle according to one aspect of the present application.

[0029] FIG. 2 is a configuration diagram of an AR system (20) according to various embodiments of the present application.

[0030] FIG. 3 illustrates a first display unit (130) and a second display unit (140) according to various embodiments of the present application.

[0031] FIG. 4 is a flowchart of a method for providing an augmented reality navigation screen for an agricultural vehicle according to another aspect of the present application.

[0032] FIG. 5 is a detailed flowchart of a process for generating an AR navigation screen in which the AR graphic object appears on the driving environment image according to various embodiments of the present application.

[0033] FIG. 6 illustrates an exemplary AR navigation screen according to various embodiments of the present application.

[0034] FIG. 7 illustrates components of a map view area according to various embodiments of the present application.

[0035] FIG. 8 illustrates an exemplary AR navigation screen displaying the rear of an agricultural vehicle according to various embodiments of the present application.

[0036] FIG. 9 is a detailed flowchart of a process for eliminating an over-the-track error (OTE) of an agricultural vehicle by providing a GUI for AB driving reference line calibration on a display according to various embodiments of the present application.

[0037] FIG. 10 illustrates an exemplary screen of an interface for calibration according to various embodiments of the present application.

[0038] FIG. 11 is a detailed flowchart of a calibration step (S830) according to various embodiments of the present application.

[0039] Figure 12 illustrates an exemplary screen displaying a baseline movement icon superimposed on the AB driving baseline.

[0040] FIG. 13 illustrates direct input of the movement distance of the reference line movement icon (740) in the calibration interface screen according to various embodiments of the present application.

[0041] FIG. 14 is a detailed flowchart of a calibration step (S830) according to various embodiments of the present application.

[0042] FIG. 15 illustrates an interface screen for calibration, in which an auxiliary moving line icon (752) and an auxiliary AB driving reference line (751) are displayed, according to various embodiments of the present application.

[0043] Fig. 16 illustrates directly inputting the movement distance of the auxiliary movement line icon (752) in the calibration interface screen of Fig. 15.

[0044] [Explanation of symbols]

[0045] 20: AR system

[0046] 110: Navigation module

[0047] 120: Memory

[0048] 130: First display section

[0049] 140: Second display section

[0050] 151: GPS module

[0051] 152: Camera module

[0052] 160: Communications Department

[0053] Hereinafter, with reference to the attached drawings, implementation examples and embodiments of the present application will be described in detail so that those skilled in the art can easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the implementation examples and embodiments described herein.

[0054] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosed form, and the scope of this specification should be understood to include various modifications, equivalents, and / or alternatives included within the technical concept.

[0055] Although terms such as "first" or "second" may be used to describe various components, such terms should be construed solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The terms "first," "second," "first," or "second," as used in various embodiments, may describe various components, regardless of order and / or importance, and do not limit the components.

[0056] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0057] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "has" should be understood to indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0058] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0059] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0060]

[0061] Agricultural machinery is considered a very important element in reducing high labor burden and production costs in response to labor shortages caused by a decreasing and aging rural workforce.

[0062] In this specification, agricultural machinery is used to perform various tasks necessary for farming, such as plowing, rotary, pesticide spraying, and rice transplanting, and is divided into a walk-behind work machine that a worker operates while walking with the agricultural machinery, and an agricultural vehicle (or passenger work machine) that a worker operates by riding in a driver's seat equipped on the agricultural machinery.

[0063] In various embodiments of the present invention, a device applicable to agricultural vehicles classified as passenger work machines such as tractors is disclosed.

[0064] The above agricultural vehicle may be coupled with a work device to drive on agricultural land such as rice paddies or fields for work. Driving the above agricultural vehicle involves such agricultural work.

[0065] The user of the above agricultural vehicle can directly perform manual driving control. The user can directly control the forward or reverse movement of the agricultural vehicle by operating the shuttle lever of the agricultural vehicle, and can control the path of the agricultural vehicle by operating the steering wheel.

[0066] In various embodiments of the present application, the agricultural vehicle may be a non-autonomous agricultural vehicle that is entirely dependent on an operator for driving functions, or an autonomous agricultural vehicle that performs the driving functions at least partially by a pre-installed autonomous driving program (or semi-autonomous driving program).

[0067] If the agricultural vehicle is an autonomous driving agricultural vehicle, the agricultural vehicle may be equipped with a driving control device to control forward or reverse driving for autonomous driving of the agricultural vehicle. In this case, the agricultural vehicle moves along a set route and can move forward or backward according to settings set by the user of the agricultural vehicle. Autonomous driving can be controlled and set through information exchange between the driving control device installed in the agricultural vehicle and a user device. The user can generate autonomous driving-related setting information and transmit it to the driving control device using a user device, such as a smartphone or computer, which is a terminal capable of information computation and information transmission and reception. The driving control device includes a transceiver and a processor, and is electrically connected to the driving unit of the agricultural vehicle to control the driving unit. The driving control device can receive autonomous driving-related setting information from the user device and control the forward or reverse movement of the driving unit of the agricultural vehicle according to the autonomous driving-related setting information.

[0068] For autonomous driving of an agricultural vehicle, a steering wheel control device may be additionally installed in the agricultural vehicle to control the steering wheel. In this case, the steering wheel control device may be configured to control the steering wheel so that the agricultural vehicle can move along a set route based on autonomous driving-related setting information received from a user device. The steering wheel control device may be mechanically connected to the steering wheel and configured to rotate the steering wheel along the set route. The steering wheel control device may include a transmitter and receiver to receive autonomous driving-related setting information from the user device. The steering wheel control device may include a processor to control the rotation of the steering wheel based on the autonomous driving-related setting information.

[0069] Hereinafter, a system and method for providing an AR navigation screen for an agricultural vehicle will be described in more detail, using examples of autonomous agricultural vehicles for explanation purposes. However, the agricultural vehicles in the embodiments according to the system and method are not limited to the autonomous agricultural vehicles, and it will be readily apparent to those skilled in the art that the agricultural vehicles may also include non-autonomous agricultural vehicles capable of generating and outputting augmented reality objects using augmented reality technology and matching the GPS location of the agricultural vehicle with the location of the agricultural vehicle recognized on a map.

[0070]

[0071] FIG. 1 is a network environment diagram of a system (10) that provides an AR navigation screen for an agricultural vehicle according to one aspect of the present application.

[0072] Referring to FIG. 1, the system (10) for providing a navigation service to the agricultural vehicle may include a system (hereinafter, AR system, 20) for providing an AR navigation screen installed in the agricultural vehicle, a network computing device (30) connected to the AR system (20) to transmit / receive data through a communication network, and an artificial satellite (40). In addition, in some embodiments, the system (10) for providing a navigation service to the agricultural vehicle may further include a user terminal (100).

[0073] The above network computing device (30), satellite (40) and user terminal (100) can also be connected to transmit / receive data through a communication network.

[0074] The network may be implemented as one or more types of any data communication network, such as 3G, 4G, 5G, LAN, WAN, Internet, telephone network, cable network, peer-to-peer network, mesh network, satellite communication, or other wired or wireless communication network.

[0075] The agricultural machinery on which the AR system (20) is installed may be a passenger agricultural machinery, and may be various agricultural vehicles such as tractors. For example, the agricultural vehicle (20) may perform agricultural or civil engineering work while towing a work machine. The agricultural vehicle (20) may provide strong traction to tow heavy loads and may provide multiple transmission gears to perform various tasks. In addition, the work machines may include various agricultural tools such as a hoe, a plow, a harrow, a rake, a rotavator, and a harvester. Depending on the type of work machine coupled to the agricultural vehicle (20), the agricultural vehicle (20) may perform various agricultural tasks such as tillage, tillage work, pest control, water pumping, and threshing.

[0076] As described above, the agricultural machinery equipped with the AR system (20) may be autonomous agricultural machinery configured to perform autonomous driving or autonomous driving operations that assist the operation of an operator (i.e., user). If the agricultural machinery equipped with the AR system (20) is autonomous agricultural machinery, the AR system (20) may be implemented in conjunction with an autonomous driving control system or integrated into a single system.

[0077]

[0078] The AR system (20) according to the embodiments may be entirely hardware, entirely software, or may have aspects that are partially hardware and partially software. For example, a device (or system) may collectively refer to hardware equipped with data processing capabilities and operating software for driving the same. In this specification, terms such as "unit," "system," and "device" are intended to refer to a combination of hardware and software driven by the hardware. For example, hardware may be a data processing device including a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other processor. In addition, software may refer to a running process, an object, an executable, a thread of execution, a program, etc.

[0079] The AR system (20) includes at least one processor capable of processing data, a memory for storing data, and a communication unit for transmitting / receiving data. The AR system (20) can acquire a map image including at least one map element from a network computing device (30) or a satellite (40). The map image can express a geographic area through the map element.

[0080] The AR system (20) may be configured to perform a navigation operation, such as setting a driving guidance route (e.g., an autonomous driving route) for an agricultural vehicle on an acquired map image and guiding the agricultural vehicle along the set driving guidance route. To this end, the AR system (20) may include a navigation application capable of performing the navigation operation, or may be connected to a third-party system providing a navigation service. For example, the AR system (20) may include a navigation API (application programming interface) for accessing a third-party system providing a navigation service.

[0081] The AR system (20) is configured to generate an AR navigation screen that displays driving-related information for agricultural machinery using augmented reality (AR) technology, and to provide the generated AR navigation screen to a user. Navigation information and driving-related information for agricultural machinery can be provided to the user through the AR navigation screen.

[0082] In addition, in various embodiments of the present application, the AR system (20) may be configured to accurately display the actual location of the agricultural vehicle on the map image. If the location of the agricultural vehicle is not accurately displayed, a discrepancy inevitably occurs between the location of the agricultural vehicle displayed on the AR navigation screen based on GPS location information and the actual location. This discrepancy goes beyond simply providing an incorrect location of the agricultural vehicle, and may also provide various related information derived from the location of the agricultural vehicle (e.g., work result area, etc.) with incorrect values. To address this, in the system 10 providing an AR navigation screen, the AR system (20) may be configured to provide a graphical user interface (GUI) capable of accurately displaying and calibrating the location of the agricultural vehicle on the map image to a first display device providing the AR navigation screen, a second display device different from the first display device, or a user terminal (100). Specifically, the user can calibrate the position of the agricultural vehicle to be accurately identified using the AB driving reference line displayed on the map and map image displayed on various display devices.

[0083] Although the AR system (20) is illustrated in this application as being a separate device from the agricultural vehicle, it will be readily apparent to those skilled in the art that in alternative embodiments, the AR system (20) may be included in or an attached device to the agricultural vehicle.

[0084] The configuration and operation of this AR system (20) are described in more detail with reference to FIGS. 3 to 16 below.

[0085]

[0086] The network computing device (30) may include a local server or a cloud server that provides a service of the system (10) providing an AR navigation screen for an agricultural vehicle according to various embodiments of the present application. The network computing device (30) may be programmed to perform an operation of providing an AR navigation screen for an agricultural vehicle and may provide a dedicated application installed in the AR system (20) or a map corresponding to the location of the agricultural vehicle to the AR system (20). Image data for the map of the network computing device (30) may be received from a satellite (40).

[0087] In some embodiments, when the agricultural vehicle is an autonomous driving agricultural vehicle, the network computing device (30) may be a server capable of providing autonomous driving-based services. To this end, the network computing device (30) may provide autonomous driving services through its own autonomous driving program, or may be connected to a server operated by an external autonomous driving company. The server (30) may transmit autonomous driving-related information of the agricultural vehicle (20) to the agricultural vehicle (20) and the user terminal (100) via a communication network, and may receive information from the agricultural vehicle (20) and the user terminal (100) via the communication network.

[0088]

[0089] The artificial satellite (40) may be a device that provides data necessary for real-world artificial satellite maps or GPS positioning, etc., required for the service of the system (10) that provides an AR navigation screen for agricultural vehicles according to various embodiments of the present application.

[0090]

[0091] The user terminal (100) may be a computing system that includes hardware, software, or embedded logic components, or a combination of two or more of these components, and can perform appropriate functions implemented or supported by the device. The user terminal (100) may be implemented in the form of a computer system, such as, for example, a desktop computer, a laptop computer, a netbook, a tablet computer, an e-book reader, a GPS device, a camera, a personal digital assistant (PDA), a portable electronic device, a cellular phone, a smart phone, other computing devices, other mobile devices, other wearable devices, other appropriate electronic devices, or any appropriate combination thereof.

[0092] Users can remotely receive AR navigation screens, navigation information displayed on the AR navigation screen, driving-related information, and autonomous driving information via a user terminal (100). Furthermore, users can control the autonomous driving operations of agricultural vehicles by inputting autonomous driving instructions via the user terminal (100).

[0093]

[0094] FIG. 2 is a configuration diagram of an AR system (20) according to various embodiments of the present application.

[0095] Referring to FIG. 2, the AR system (20) includes a navigation module (110), a memory (120), a first display unit (130), a second display unit (140), a GPS module (151), a camera module (152), and a communication unit (160). In some embodiments, the AR system (20) may further include a steering sensor (153), an autonomous driving sensor (154), and / or a driving sensor (155).

[0096] The navigation module (110) may be implemented with at least one processor. The processor may include any combination of a CPU, a GPU (graphical processing unit), a single-core processor, a multi-core processor, an ASIC (application specific integrated circuit), and the like. In addition to a hardware implementation, the processor may be implemented with software and / or firmware. The software or firmware implementation of the processor may include computer- or machine-executable instructions written in any suitable programming language to perform the various functions described above. The software implementation of the processor may be stored in whole or in part within the memory (120).

[0097] The above navigation module (110) may be configured to generate an AR navigation screen that expresses at least one of navigation information, driving-related information, and driving error information as an AR graphic object. The process by which the navigation module (110) generates the AR navigation screen is described in more detail with reference to FIGS. 4 and 5 below.

[0098] The memory (120) can store programs of instructions that can be loaded and executed on the processor, and data generated during the execution of these programs. Examples of programs and data stored on the memory (120) may include an operating system that controls the operation of hardware and software resources available in the AR system (20), drivers for interacting with hardware devices such as agricultural vehicles and network computing devices (30), communication protocols that can exchange data with other hardware devices such as agricultural vehicles and network computing devices (30), and additional software applications. The memory (120) may be volatile (such as RAM) or non-volatile (such as ROM or flash memory). The memory (120) may provide storage of computer-readable instructions, data structures, program modules, and other data. The computer-readable medium may include at least two types of computer-readable media, namely, computer storage media and communication media.

[0099] Additionally, the memory (120) can store a map image and map information corresponding to the map image.

[0100] FIG. 3 illustrates a first display unit (130) and a second display unit (140) according to various embodiments of the present application.

[0101] Referring to FIG. 3, the first display unit (130) is an output device that displays an AR navigation screen. The second display unit (140) can display user input for controlling an agricultural vehicle, control results, work results, vehicle status, and various contents provided by the infotainment system.

[0102] In various embodiments of the present application, the second display unit (140) may include an output device (141) and an input device (142).

[0103] The output devices (141) of the first display unit (130) and the second display unit (140) may be implemented as liquid crystal displays (LCDs), organic light emitting diodes (OLEDs), active matrix organic light emitting diodes (AMOLEDs), etc. The display units (130, 140) visually provide data stored in the AR system (20) or input data to the user.

[0104] In various embodiments of the present application, the second display unit (140) may display an AB driving reference line on a map and a map image by executing at least one application, and may provide a GUI in which various objects, modals, and indicators for calibrating the AB driving reference line are displayed.

[0105] The input device (142) is a component configured to receive a command related to a user's input. The input device (142) may include a touch unit or other input units. The touch unit is a component that inputs a user command by using a part of the user's body or another object as a pointing object. The touch unit may include, but is not limited to, a pressure-sensitive or electrostatic sensor such as a capacitive overlay or a resistive overlay. In addition to the sensor, the touch unit may be implemented with any type of sensor device capable of detecting the contact of an object. The touch sensor (142) may detect a user's touch input, generate a detection signal, and transmit the detection signal to the processor. The detection signal detected by the touch sensor (142) may include coordinate data of the touch input by the user.

[0106] The above-described other input units include, for example, buttons, keyboards, dials, switches, sticks, keys, etc. In one embodiment of the present application, the AR system (20) may further include a physical button in addition to the touch sensor (132). The physical button may be, for example, a button that receives user input for switching driving modes, such as turning autonomous driving mode on / off or manual driving mode on / off.

[0107] Although the input device (142) and the output device (141) are separated in FIG. 3, in many embodiments, the input device (142) and the output device (141) may be implemented as a single component to perform input reception and information output. For example, the input device (142) and the output device (141) may be a touch panel implemented as a touch screen that forms a screen and layer structure. Touch input is input by a pointing object (e.g., a user's body, a tool, etc.).

[0108] In this application, the first display unit (130) and the second display unit (140) are illustrated as separate devices, but it will be clearly understood by those skilled in the art that in alternative embodiments, the first display unit (130) and the second display unit (140) can be implemented as a single device.

[0109] The GPS module (151) can obtain GPS location information indicating the GPS coordinates of the module (151) by communicating with GPS satellites. Since the module (151) is installed in an agricultural vehicle, the GPS location information is treated as indicating the location of the agricultural vehicle. The GPS module (151) can transmit the GPS location information of the agricultural vehicle to the navigation module (110) to be used for generating driving-related information such as the real-time location, real-time driving speed, and driving time of the agricultural vehicle.

[0110] The camera module (152) can capture driving environment images representing the environment in which the agricultural vehicle is driving, thereby generating driving environment image data. The driving environment images may be images of objects located in front, behind, or around the agricultural vehicle. The objects may be a user or another person, farmland on which agricultural work is to be performed, crops that are the target of agricultural work, obstacles located along the path, etc.

[0111] The above camera module (152) may be a variety of imaging devices that recognize light and generate images.

[0112] The driving environment image may be a captured image or a video image composed of continuous driving environment images. The camera module (152) may transmit the real-time driving environment image to the navigation module (110). For example, the camera module (152) may transmit a real-time driving environment video to the navigation module (110).

[0113] The AR system (20) may include one or more camera modules (152). In various embodiments of the present application, the AR system (20) may include at least one camera module (152a) capable of photographing the front of an agricultural vehicle, and at least one camera module (152b) capable of photographing the rear of the agricultural vehicle. The driving environment image photographed by the camera module (152a) is a front-view image representing the front driving environment, and the driving environment image photographed by the camera module (152)b is a rear-view image representing the rear driving environment.

[0114] The steering angle sensor (153) is a sensor that detects the steering angle of the steering wheel. The steering angle sensor (153) may be a current sensor that measures the rotation angle in response to electrical characteristics that change according to the rotation of the steering wheel, or an angle sensor that measures the degree of rotation of the steering wheel, but is not limited thereto.

[0115] The autonomous driving sensor (154) may be a sensor used to recognize objects located around an agricultural vehicle. The autonomous driving sensor (154) may be, for example, a radar, a Lidar, an ultrasonic sensor, or other sensors.

[0116] The driving sensor (155) is a sensor that measures the real-time driving speed of the vehicle. The driving speed of the vehicle measured by the driving sensor (155) can be displayed on the dashboard of the agricultural vehicle.

[0117] In addition, the agricultural vehicle 20 and the remote user terminal (100) can receive GPS coordinates of the agricultural vehicle acquired through the GPS module (151) installed in the agricultural vehicle through wireless communication with the AR system (20).

[0118]

[0119] The wireless communication unit (160) can perform a wireless data transmission and reception function of the AR system (20). The wireless communication unit (160) can establish wireless communication with an agricultural vehicle and / or a network computing device (30).

[0120] The wireless communication unit (160) may include, for example, a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module. The wireless communication unit (160) may establish a communication connection with an agricultural vehicle or a network computing device (30) via a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN).

[0121] A system (10) providing an AR navigation screen for an agricultural vehicle according to various embodiments of the present application can provide an AR system (20) with a graphic user interface (GUI) that can accurately display and calibrate the location of an agricultural vehicle on a map image.

[0122] In the AR system (20) according to various embodiments of the present application, an application (hereinafter referred to as an application) may be executed to output an AR navigation screen including navigation information including a driving guidance path set for the operation of an agricultural vehicle, driving-related information acquired while the agricultural vehicle is driving along the driving guidance path, and driving error information indicating the gap between the actual location of the agricultural vehicle and the GPS location, and to output a graphic user interface (GUI) capable of accurately displaying and calibrating the location of the agricultural vehicle on a map image. The application may be configured to perform a method for providing an augmented reality navigation screen for an agricultural vehicle described with reference to FIG. 4 below.

[0123] In various embodiments, the application may be provided as included in at least one computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™ or App Store™), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0124]

[0125] A method for providing an augmented reality navigation screen for an agricultural vehicle according to another aspect of the present application can be performed by the AR system (20) of FIG. 2.

[0126] FIG. 4 is a flowchart of a method for providing an augmented reality navigation screen for an agricultural vehicle according to another aspect of the present application.

[0127] Referring to FIG. 4, the method for providing an AR navigation screen for the agricultural vehicle includes a step (S100) of obtaining GPS location information describing the GPS location of the agricultural vehicle; a step (S200) of obtaining navigation information set from an external device or navigation information set according to a user input to the AR system; and a step (S300) of obtaining a driving environment image capturing a driving environment in which the agricultural vehicle is driving. In addition, the method for providing the AR navigation screen may further include a step of obtaining a user input for a driving mode indicating on / off of an autonomous driving mode prior to the step (S500).

[0128] In the above step (S100), the navigation module (110) can obtain GPS location information of the target agricultural vehicle on which the AR system (20) is installed from the GPS module (151). The GPS location information is the location of the agricultural vehicle recognized by GPS satellites, and may differ from the actual location of the agricultural vehicle depending on GPS performance.

[0129] The above GPS location information may include information on the time at which the GPS coordinates of the agricultural vehicle were detected. The acquired GPS location information may be stored in memory (120) (S100).

[0130] In the step (S200), the navigation information includes driving guidance route information that describes a driving guidance route along which the agricultural vehicle will drive. In addition, the navigation information may further include one or more pieces of information from a map image for the driving guidance route, map information for the map image, and a driving guidance message for the driving guidance route, which indicate a geographic area in which the driving guidance route is located. The map image may visually represent a geographic area around the GPS coordinates of the agricultural vehicle. The map information describes map elements shown in the map image. The map elements describe the driving environment of the agricultural vehicle, and may include, for example, real estate such as buildings, land, and roads, and their sizes, locations, names, slopes, ground / building types, and topographic features such as mountains and rivers, and their altitudes, sizes, locations, names, slopes, types, etc. In addition, the map elements may further include information describing real estate such as crops and trees, or objects located on topographic features.

[0131] The above driving guidance route information indicates the starting point, destination, and the location of the route connecting them. The driving guidance route includes both the previously driven route and the planned future route.

[0132] In the above step (S200), the navigation module (110) may acquire any one of external navigation information set from a third-party system providing an external navigation service, internal navigation information set through its own navigation application based on user input to the AR system (20), and a combination thereof as navigation information for generating the AR navigation screen. For example, the navigation information may acquire a map image and map information through a network computing device (30), and acquire driving route information through its own navigation application.

[0133] In the above step (S300), the navigation module (110) can obtain a driving environment image of the agricultural vehicle through the camera module (152). The driving environment image may be a real-time video image that captures the surrounding environment of the agricultural vehicle in real time.

[0134] The navigation module (110) can obtain an image of the surrounding environment in front of the agricultural vehicle from the camera module (152a) and an image of the surrounding environment in the rear of the agricultural vehicle from the camera module (152b).

[0135] In addition, the method includes a step (S500) of calculating driving-related information of the agricultural machine based on GPS location information of the agricultural machine. In addition, in some embodiments, when the AR system (20) further includes a steering angle sensor (153) and / or a driving sensor (155), a step (S400) of obtaining steering angle information and / or real-time driving speed information may be further included prior to step (S500).

[0136] When the AR system (20) further includes a steering angle sensor (153) and / or a driving sensor (155), the navigation module (110) can obtain steering angle information received from the steering angle sensor (153) and / or driving speed information received from the driving sensor (155) (S400). This information represents the real-time steering angle and real-time driving speed at the time of measurement. The navigation module (110) can use the steering angle information and / or driving speed information to generate driving-related information to be displayed on the AR navigation screen (S500).

[0137] The above navigation module (110) can collect GPS location information of the agricultural vehicle acquired in step (S100) while the agricultural vehicle is driving. For example, the navigation module (110) can collect GPS location information acquired from the starting point of the driving guidance route to the current location at the time of generating the AR navigation screen.

[0138] The above navigation module (110) can calculate driving-related information of the agricultural vehicle based on GPS location information of the agricultural machine acquired while the agricultural machine is driving, or based on GPS location information and steering angle information of the agricultural machine acquired while the agricultural machine is driving (S500).

[0139] In various embodiments of the present application, the driving-related information may include one or more of driving mode information, driving route, driving distance according to the driving route, driving time, real-time driving speed of the agricultural vehicle at the real-time location of the agricultural vehicle, driving direction angle, and work result area.

[0140] Driving mode information represents user input regarding driving modes. The user input indicates the on / off status of autonomous driving mode and the setting range for the autonomous driving speed limit (e.g., maximum speed limit, etc.).

[0141] A driving route is a tracked path that represents the actual driving of an agricultural vehicle along a driving guidance route. The navigation module (110) can calculate a driving route by tracking changes in the GPS location of the agricultural machine over time, acquired while the machine is driving. The driving route may be the result of driving for a single agricultural operation.

[0142] The driving distance and driving time for a driving route are the distance traveled and the time consumed from the driving start point to the current location at the time of generating the AR navigation screen. The navigation module (110) can calculate the driving distance and driving time based on the calculated driving route, driving start time, and the current time at the time of generating the AR navigation screen.

[0143] The driving direction angle represents the angle at which the vehicle rotates relative to the vehicle's plan view. The driving direction angle can be expressed in a range of 360° (e.g., 0° to 180° and -180° to 0°). The navigation module (110) can calculate the driving direction angle based on changes in GPS location information or steering angle information.

[0144] The work result area is an area that shows the results of work performed by an agricultural vehicle while passing through a driving path. The work result area information may indicate the location and area of ​​the area. The navigation module (110) may identify the work result area by applying the driving path calculated to the unit work area per unit distance stored in advance. Then, the specifications of the unit work area may be accumulated based on the location of the driving path to define the work result area. The area of ​​the work result area may be referred to as the driving area as the area of ​​the identified work result area. In one example, the specifications of the unit work area may be a value matching the width of the #agricultural vehicle. In this case, the driving area of ​​the agricultural vehicle may be the work result area of ​​the agricultural vehicle.

[0145] In addition, the method includes a step (S600) of obtaining the difference between the driving guidance path (or location) and the actual driving path (or location) of the agricultural machine as driving error information of the agricultural machine.

[0146] The above driving error information may be referred to as Offset Error (OTE). In agricultural operations, OTE is a term used to describe the difference between the expected path or location and the actual path or location, primarily using GPS or a positioning system. It is used as a value indicating how much the actual path deviates from the expected path. The driving error information indicates the location of the driving error, the distance between the driving error points, or the area of ​​the driving error area.

[0147] The above navigation module (110) can calculate the difference between the driving guidance path and the actual driving path of the agricultural vehicle through user input or an error calculation program.

[0148] The navigation module (110) can obtain the difference between a location on the driving guidance route and a location on the actual driving route of the agricultural vehicle (S600). The locations of each route are locations at the same point in time. If the accuracy of the GPS location information is not 100%, the difference between them will not be zero even if it is very small. An error of at least several centimeters and at most several meters may occur between the location of the agricultural vehicle (20) indicated by the GPS coordinates and the actual location of the agricultural vehicle (20).

[0149] The difference between the above-mentioned driving guidance path and the actual driving path of the agricultural vehicle can be calculated by calculating the distance between different points in a geometric relationship, or by calculating the distance between the above-mentioned driving guidance path and the actual driving path of the agricultural vehicle. The above-mentioned navigation module 600 can obtain the driving error information of the agricultural vehicle by calculating the difference between the above-mentioned driving guidance path and the actual driving path of the agricultural vehicle (S600).

[0150] In addition, the method includes a step (S700) of generating at least one graphic object that visually expresses at least one of the navigation information, driving-related information of the agricultural machinery, and driving error information, and generating an AR navigation screen in which the AR graphic object appears on the driving environment image.

[0151] FIG. 5 is a detailed flowchart of a process for generating an AR navigation screen in which the AR graphic object appears on the driving environment image, according to various embodiments of the present application. FIG. 6 illustrates an exemplary AR navigation screen, according to various embodiments of the present application.

[0152] The above AR navigation screen may include multiple AR graphic objects. The AR graphic objects are virtual three-dimensional components that are added to the real environment to provide visual information or experiences to the user.

[0153] Referring to FIG. 5, the step (S700) of generating the AR navigation screen includes the steps of: identifying the position of a graphic object representing at least one of the navigation information and the driving-related information in the driving environment image (S710); generating the shape of each graphic object based on the identified position (S720); and overlaying the shape of the graphic object with the identified color applied to the position on the identified driving environment image to generate an AR navigation screen in which the shape of the graphic object is expressed in the driving environment image (S740). In addition, in some embodiments, the step (S700) may further include the step (S730) of applying a preset color and / or transparency for the information represented by the graphic object to the shape of the graphic object prior to the step (S740).

[0154] In various embodiments of the present application, the AR graphic object includes a first graphic object (301) that visually represents a driving guidance path. In addition, the AR graphic object may further include one or more graphic objects among a second graphic object (302) that visually represents a work result area according to driving of the agricultural vehicle, a third graphic object (303) that visually represents a driving direction, and a fourth graphic object (304) that visually represents the driving error information.

[0155] In the above step (S710), the navigation module (110) can identify the location on the two-dimensional coordinate system on the driving environment image where each AR graphic object (301 to 304) is to be placed.

[0156] In various embodiments of the present application, the step (S710) of identifying the position of the AR graphic object on the driving environment image includes: a step of identifying the position of the first graphic object (301) based on GPS position information of the agricultural vehicle, map information for a geographical area including the GPS position of the agricultural vehicle, driving direction information of the agricultural vehicle, and parameters of the camera module; a step of identifying the position of the second graphic object (302) based on the position of the identified first graphic object (301), driving path information, driving distance information, and unit work result area information per unit driving distance stored in advance; a step of identifying the position of the third graphic object (303) based on the position and driving direction information of the identified first graphic object (301) when the AR graphic object includes the third graphic object (303); If the AR graphic object includes the fourth graphic object (304), the method may include a step of identifying the position of the fourth graphic object (304) based on the position of the identified first graphic object (301) and the driving error information.

[0157] The navigation module (110) can define the relationship between a two-dimensional image of the driving environment, a GPS location, and an AR graphic object by calibrating the internal and external parameters of the camera to identify the location of the AR graphic object.

[0158] The above navigation module (110) can identify the location of the first graphic object (301) where the first graphic object (301) is to be placed on the driving environment image based on the current GPS location information of the agricultural vehicle at the time of generating the AR navigation screen, map information for a map image including coordinates where the agricultural vehicle is located, current driving direction information of the agricultural vehicle at the time of generating the AR navigation screen, and parameters of a camera module.

[0159] In various embodiments of the present application, the step of identifying the position of the first graphic object (301) may include: a step of mapping a two-dimensional coordinate system in the driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world; a step of calculating a three-dimensional position of the driving guidance path in the virtual three-dimensional world based on the map information and the driving guidance path information; and a step of changing the three-dimensional position of the driving guidance path to a two-dimensional position of the driving guidance path through a mapping relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, and identifying the changed two-dimensional position of the driving guidance path as the position of the first graphic object.

[0160] The above two-dimensional coordinate system is a two-dimensional coordinate system in the driving environment image. The above virtual three-dimensional world is a three-dimensional virtual space for the geographical area shown in the map image. The navigation module (110) maps the two-dimensional coordinate system for the driving environment image and the three-dimensional coordinate system in the virtual three-dimensional world, thereby defining a positional relationship (hereinafter, “coordinate mapping relationship”) between the two-dimensional coordinate system and the virtual three-dimensional world. Then, the navigation module (110) can convert the virtual three-dimensional coordinates into two-dimensional coordinates in the driving environment image using the defined coordinate mapping relationship.

[0161] In some embodiments, the step of mapping a two-dimensional coordinate system in the driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world includes a step of setting internal parameters of the camera module (152) to map two-dimensional coordinates of the driving environment image and coordinates of the virtual three-dimensional world. Furthermore, in some embodiments, the step of mapping a two-dimensional coordinate system in the driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world may further include a step of analyzing a relationship between an object in the three-dimensional world and a two-dimensional representation of the object in the driving environment image in which the object is photographed, and setting external parameters of the camera module (152) based on the virtual three-dimensional world to which coordinates are mapped.

[0162] The navigation module (110) may perform an operation of optimizing the internal parameters of the camera module (152) to map the two-dimensional coordinates of the driving environment image and the virtual three-dimensional world coordinates. The intrinsic parameters of the camera module (152) include one or more of a principal point, a focal length, and a distortion coefficient. The principal point is a point on the image sensor where the center line of the lens passes through the sensor, and in most cases, is located close to the center of the image. The focal length is the distance between the lens and the image sensor. The focal lengths for the x-axis and the y-axis may be different. The distortion coefficient is a coefficient for correcting the deformation of the image caused by lens distortion. It can be mainly expressed as a coefficient for radial distortion and tangential distortion.

[0163] The external parameters of the above camera module (152) may further include one or more of rotation and translation. The rotation parameter indicates the rotation direction of the camera and may be expressed as a 3x3 rotation matrix. The translation parameter may be expressed as a translation vector from the world coordinate system to the camera coordinate system.

[0164] The navigation module (110) may analyze the relationship between the features of an object in the 3D world and the 2D representation of the features of the object that appear in the driving environment image in which the object is photographed, in order to analyze the relationship between the features of the object in the 3D world and the 2D representation of the features of the object that appear in the driving environment image in which the object is photographed. The navigation module (110) may extract the 2D representation through an image processing algorithm that extracts the 2D representation, and analyze the 2D representation of the object and the 3D features of the object. The features may be geometric features such as edges, corners, etc.

[0165] Objects on the above 3D world are objects used as map elements, and may be, for example, buildings, terrain features (e.g., mountains), crops, etc.

[0166] The driving environment image providing the above two-dimensional expression may be a continuous driving environment image (e.g., rotation video, movement video) captured during the process of rotating or moving the camera module (152).

[0167] In some embodiments, the navigation module (110) may map a three-dimensional coordinate system and a two-dimensional coordinate system using the GPS location of the agricultural vehicle. For example, an object providing a two-dimensional representation may be an object located around the GPS location of the agricultural vehicle.

[0168] In some embodiments, the step (S710) may further include a step of recognizing an object located around the agricultural vehicle by analyzing an object appearing in the driving environment image.

[0169] The above navigation module (110) can obtain a positional relationship between a position of an object recognized in a two-dimensional driving environment image and a three-dimensional position of the object in order to map a two-dimensional coordinate system and a three-dimensional coordinate system, and map two-dimensional coordinates of the driving environment image and virtual three-dimensional world coordinates based on the positional relationship and the setting results of internal parameters, thereby calculating a coordinate mapping relationship.

[0170] The navigation module (110) can recognize objects using various object recognition models. The object recognition model may be a vision recognition model that extracts features from an input image, performs computational processing on them, and infers a correlation between the object and the extracted features based on the computational processing results to recognize the object in the input image. The object recognition model may have a CNN structure, but is not limited thereto. For example, the object recognition model may be an open source vision recognition model such as OpenCV.

[0171] The above navigation module (110) can identify the positions of the remaining graphic objects (302 to 304) based on the position of the first graphic object (301) identified in the two-dimensional coordinate system for the driving environment image (S710).

[0172] The location of the work result area corresponding to the second graphic object (302) may be the location of points and / or lines defining the work result area. In some embodiments, the location of the second graphic object (302) may be expressed as the location of the driving path and the unit work result area per unit driving distance (e.g., unit working width). The navigation module (110) calculates the three-dimensional location of the work result area based on the work result area information of the agricultural vehicle, which is calculated in advance through the location of the identified first graphic object, driving path information, driving distance information, and the unit per unit driving distance stored in advance, in order to identify the location of the second graphic object (302), and changes the three-dimensional location of the work result area into a two-dimensional location on a two-dimensional coordinate system for the driving environment image of step (S300) through a coordinate mapping relationship, and can identify the two-dimensional location of the changed work result area as the location of the second graphic object. The three-dimensional location is a location on the virtual three-dimensional world.

[0173] The position of the current driving direction corresponding to the third graphic object (303) may be the position of a virtual line extended by the driving direction angle from the current position of the agricultural vehicle. The navigation module (110) calculates a three-dimensional position for a virtual line indicating the current driving direction at the time of generating the AR navigation screen based on the position and driving direction information of the identified first graphic object (301), in order to identify the position of the third graphic object (303), and changes the three-dimensional position for the current driving direction into a two-dimensional position on the two-dimensional coordinate system for the driving environment image of the step (S300) through the coordinate mapping relationship, and can identify the changed two-dimensional position of the current driving direction as the position of the third graphic object (S303).

[0174] The location of the driving error corresponding to the fourth graphic object (304) may be the location of an interval indicating the driving error. If the interval extends along the driving path, the driving error may form a driving error area having an integrated area due to the continuous interval. Then, the location of the driving error corresponding to the fourth graphic object (304) may be the location of the driving error area, and may be the location of points and / or lines defining the driving error area. The navigation module (110) calculates a three-dimensional position for the driving error based on the position of the identified first graphic object (301) and the driving error information to identify the position of the fourth graphic object (304) based on the position of the identified first graphic object (301) and the driving error information, and changes the three-dimensional position for the driving error into a two-dimensional position on the two-dimensional coordinate system for the driving environment image of the step (S300) through the coordinate mapping relationship, and can identify the changed two-dimensional position of the driving error as the position of the fourth graphic object (304). The position of the fourth graphic object (304)

[0175] In the above step (S720), the navigation module (110) can apply preset shape setting information for each AR graphic object (301 to 304) to an identified location with respect to the two-dimensional coordinate system of the driving environment image, thereby generating the shape of each AR graphic object (301 to 304). The shape of the AR graphic object is a shape expressed in the driving environment image.

[0176] In some embodiments, the shape setting information for the first graphic object (301) may represent a line shape that extends along a location of a driving guidance path. The navigation module (110) may apply the shape setting information for the first graphic object (301) to an identified location of the first graphic object (301) to generate a line shape that extends along the identified location as the shape of the first graphic object (301) to be displayed on a two-dimensional driving environment image.

[0177] The shape setting information for the second graphic object (302) may represent an area shape that covers the work result area. The navigation module (110) may apply the shape setting information for the second graphic object (302) to the identified location of the second graphic object (302) to generate an area shape that covers the identified location as the shape of the second graphic object (302) to be displayed on a two-dimensional driving environment image.

[0178] The shape setting information for the third graphic object (303) may indicate at least one continuous arrow shape along the driving direction. The navigation module (110) may apply the shape setting information for the third graphic object (303) to the identified position of the third graphic object (301) to generate a continuous arrow shape along the identified position as the shape of the third graphic object (303) to be displayed on a two-dimensional driving environment image.

[0179] The shape setting information for the fourth graphic object (304) may represent a line shape when the driving error information is a driving error interval, and may represent an area shape when the driving error information is a driving error area. For example, the navigation module (110) may apply the shape setting information for the fourth graphic object (304) to the identified location of the fourth graphic object (304) to generate an area shape covering the identified location as the shape of the fourth graphic object (304) to be displayed on a two-dimensional driving environment image.

[0180] In the above step (S730), the navigation module (110) may apply preset color and / or transparency setting information for each AR graphic object to the shape of the AR graphic object. For example, as illustrated in FIG. 6, the color setting information for the driving guidance route may be white, and the color setting information for the driving direction may be green, but is not limited thereto.

[0181] In some embodiments, the color setting information for the work result area may include color setting information according to the driving mode. Specifically, the color setting information for the work result area may include first color setting information for the work result area determined by driving according to the manual driving mode (i.e., the off state of the autonomous driving mode), and second color setting information for the work result area determined by driving according to the autonomous driving mode (i.e., the on state of the autonomous driving mode). For example, as illustrated in FIG. 6, the first color setting information may be orange, and the second color setting information may be green. When such color setting information is applied, as illustrated in FIG. 6, the work result area worked on in the manual driving mode (i.e., the non-autonomous driving mode) among the entire work results of the agricultural vehicle may be implemented as a second graphic object (303a) to which the first color is applied, and the work result area worked on in the autonomous driving mode may be implemented as a second graphic object (303b) to which the second color is applied, and may be displayed on the AR navigation screen.

[0182] The navigation module (110) applies a preset transparency level to each piece of information pointed to by an object. The transparency level is set to either a maximum value indicating complete transparency or a minimum value indicating complete opacity. The transparency level for at least one graphic object may be set to a partial transparency value.

[0183] In some embodiments, the transparency of the second graphic object (302) and the third graphic object (303) may be set to be more transparent than the transparency of the first graphic object (302) and the fourth graphic object (304).

[0184] When AR graphic objects (301 to 304) with preset colors applied are overlaid on the shapes generated in this manner, an AR navigation screen can be generated that can provide driving error information, driving direction information, work result area information, and driving guidance path information to the user through visual elements such as the color, location, and shape of the AR graphic objects (301 to 304) as shown in FIG. 6 (S740).

[0185] In addition, in some embodiments, the step (S700) of generating the AR navigation screen may further include a step (S760) of generating a map view area (310) that shows the recognized location of the agricultural machine based on GPS location information of the agricultural machine, a map image including a geographical area around the agricultural machine, driving direction information of the agricultural machine at the recognized location, work result area information, and an area around the agricultural machine; and / or a step (S770) of generating an error view area (320) that describes the current driving error of the agricultural machine at the time of generating the AR navigation screen based on the driving error information. In addition, the step (S700) may further include a step (S780) of additionally overlaying one or more areas of the error view area 340 of the map view area (310) on the driving environment image of the step (300) in order to generate the AR navigation screen in which one or more of the map view area and the error view area are additionally shown.

[0186] The map view area (310) and error view area (320) are composed of a number of display elements displayed in the corresponding areas. The display elements may be two-dimensional graphic objects that visually represent the corresponding information on a two-dimensional plane.

[0187] FIG. 7 illustrates components of a map view area according to various embodiments of the present application.

[0188] Referring to FIG. 7, the map view area (310) may include a driving vehicle object (315) indicating the recognized position of the agricultural vehicle, a driving guidance path object (311) indicating a driving guidance path, a driving direction object (312) indicating the driving direction of the agricultural vehicle, and a work result area object (313) indicating a work result area of ​​the agricultural vehicle.

[0189] The above-described driving vehicle object (315) may have a preset vehicle shape corresponding to an agricultural vehicle. The driving vehicle object (315) displays the location of the agricultural vehicle on a map of a navigation screen. The shape of the driving object may be a realistic shape, such as a plan view of the agricultural vehicle, or a simpler shape, such as a circle, as illustrated in FIG. 3. The driving vehicle object (315) with a simpler shape is not limited to a circle, and may also have a triangle, square, oval, etc.

[0190] The location of the agricultural vehicle indicated by the above-mentioned driving vehicle object (315) is the recognized location of the agricultural machine recognized by the AR system (20). The navigation module (110) primarily recognizes the location of the agricultural machine based on the GPS coordinates indicated in the GPS location information of the agricultural machine (S760). In addition, when the actual location of the agricultural vehicle and the GPS location are matched by the calibration operation described below to eliminate the driving error, the recognized location of the agricultural machine indicated by the above-mentioned driving vehicle object (315) represents the actual location corrected by reflecting the calibration relationship in the GPS coordinates.

[0191] The driving direction object (312) represents the measured driving direction at the time of display. The driving direction is

[0192] The driving direction object (312) may be, for example, in the shape of a single arrow as shown in FIG. 7, but is not limited thereto, and the arrow may be expressed only as a line or as a combination of a line and a shape having an area. In various embodiments, the driving direction object (312) may be included within the shape of the driving vehicle object (315).

[0193] Referring back to FIG. 6, the error view area 340 may represent a relative position of a driving error with respect to the agricultural vehicle, an interval length of the driving error, or an area area. In some embodiments, the navigation module (110) may calculate a relative position of a driving error with respect to the agricultural vehicle based on GPS location information of the agricultural vehicle and a position of the driving error within the driving error information, and may generate an error view area (340) representing the interval length of the driving error and the relative position of the driving error with respect to the agricultural vehicle, as shown in FIG. 6 (S770).

[0194] Additionally, in some embodiments, the step (S700) may further include a step (S790) of displaying at least one piece of information among the navigation information and driving-related information on the AR navigation screen. The at least one piece of information may be displayed in any one of text form, graphic form, and a combination thereof.

[0195] Specifically, the step (S790) may include a step of displaying driving distance information, area of ​​a work result area, and driving time information of the agricultural vehicle in the generated driving-related information in a first sub-area (360) of the navigation screen; a step of displaying driving mode information of the agricultural vehicle in a second sub-area (370) of the navigation screen; and / or a step of displaying the contents of a driving guidance message in the acquired navigation information in a sub-area (380) of the navigation screen.

[0196] The above navigation module (110) may be configured to display the driving distance information, the area of ​​the work result area, and the driving time information of the agricultural vehicle in the driving-related information calculated in the above step (S500) in the form of text in the sub-area (360) of the navigation screen.

[0197] In addition, the navigation module (110) may be configured to display driving mode information of the agricultural vehicle in a combination of text and graphic forms in a sub-area (370) of the navigation screen. In one example, when the current driving mode of the agricultural vehicle at the time of generating the AR navigation screen is autonomous driving, the sub-area (370) may display text (AUTO) indicating the autonomous driving mode, text (15 km / h) indicating the setting range of the autonomous driving speed limit (e.g., maximum speed limit), and a border shape indicating the on state of the autonomous driving mode.

[0198] In addition, the navigation module (110) may be configured to display the content of the driving guidance message within the navigation information acquired in the step (S200) in a sub-area (380) of the navigation screen. The graphic form displaying the content of the driving guidance message may be implemented as a graphic object summarizing the content of the driving guidance message.

[0199] The above navigation module (110) can output an AR navigation screen including graphic objects (301 to 304) overlaid in step (S740), information display areas (310, 340) overlaid in step (S780), and information (360 to 380) expressed in step (S790) through the first display unit (130).

[0200] FIG. 8 illustrates an exemplary AR navigation screen displaying the rear of an agricultural vehicle according to various embodiments of the present application.

[0201] Referring to FIG. 8, the driving environment image overlaid in the step (S740) is a driving environment image in the shooting direction specified at the time of the AR navigation screen. If the user specifies an AR navigation screen that displays a forward shooting direction, graphic objects (301 to 304) are overlaid on the driving environment image that displays the forward driving environment, thereby providing the AR navigation screen of FIG. 6. Alternatively, if the user specifies an AR navigation screen that displays a rear shooting direction, graphic objects (301 to 304) are overlaid on the driving environment image that displays the rear driving environment, thereby providing the AR navigation screen of FIG. 8.

[0202] Since the AR navigation screen of FIG. 8 includes screen components (301 to 304, 310, 340 to 380) that are identical or similar to the AR navigation screen of FIG. 6, a detailed description thereof is omitted.

[0203] In various embodiments of the present application, the AR navigation screen may be further configured to receive a user input for changing the shooting direction. When a user touch input is input into a sub-area (350) indicating the shooting direction to be changed, the AR navigation screen may generate an AR navigation screen that displays a driving environment image corresponding to the selected shooting direction in response to a user input for selecting a different shooting direction.

[0204] When a user's touch is input to the sub-area (350), the navigation module (110) may perform at least some of the operations in steps (S710 to S790) to generate an AR navigation screen for the shooting direction to which the user has commanded a change, and output the AR navigation screen through the first display unit (130). Specifically, when a user's touch is input to the sub-area (350), the navigation module (110) may perform the operations in steps (S710 to S740) in response to a touch in the sub-area (350). Here, the driving environment image in steps (S740, S780) is an image of the shooting direction selected for change (e.g., rear view).

[0205] For example, when a user touches a sub-area (350) on an AR navigation screen displaying the front of an agricultural vehicle as shown in FIG. 6, an AR navigation screen displaying the rear of an agricultural vehicle as shown in FIG. 8 can be generated and displayed on the first display unit (130). Conversely, when a user touches a sub-area (350) as shown in FIG. 8, the AR navigation screen of the first display unit (130) can be switched from the AR navigation screen of FIG. 8 to the AR navigation screen of FIG. 6.

[0206] The AR navigation screen generated in this way not only displays the OTE in the overview area (340) but also displays the corresponding AR graphic object (304) in a unique identification color (red) to clearly convey information. Through this method, the user can check the accuracy of the driving route in real time. If a radar is installed together with the camera, the navigation module (110) can detect obstacles in front and provide warning graphics and sounds to enhance safety. The driving route can be clearly distinguished by color between the driving guidance route (applied in white) and the past driving route (green: autonomous driving / orange: manual driving), so that the driving history can be accurately distinguished. Through this, in the case of a sprayer tractor equipped with a weeder, the area where weeding has been performed and the area where weeding will be performed can be accurately distinguished using the AR navigation screen, so overlapping can be avoided, thereby avoiding herbicide waste and enabling work to proceed with optimal efficiency. In addition to the real-time view from the camera perspective, a map view (310) is provided, allowing the location and route of the currently operating vehicle to be identified from a macroscopic perspective. Since the driving route (311) and the passed route (313a, 313b) are displayed identically in the map view (310), they can be compared with the real-time AR route, enabling accurate location identification. In the speedometer, when autonomous driving is activated, a circular graphic outline (360) is displayed in a unique color (e.g., green), and when switched to manual driving, it is displayed in orange. This allows the current driving status to be checked while minimizing gaze movement without having to look away from another screen.

[0207]

[0208] Referring again to FIG. 4, in various embodiments of the present application, the method may further include a step (S800) of removing a driving error using the AB driving reference line.

[0209] When an agricultural vehicle (20) must autonomously drive along a precise path, such as a tractor planting seeds, an error of even 10 cm may not be acceptable. The navigation module (110) can perform a calibration operation to match the GPS coordinates of the agricultural vehicle with its actual location using the AB driving reference line (S800). Specifically, the navigation module (110) can provide a GUI for AB driving reference line calibration on the display to eliminate driving error information (S800).

[0210] The GUI screen for the above AB driving reference line calibration may be displayed on a map image on a second display installed in the agricultural vehicle or on a user terminal (100) communicating with the agricultural vehicle.

[0211] By the operation of the above step (S800), the fourth graphic object (304) displayed on the AR navigation screen of FIGS. 6 and 7 may be relatively reduced compared to before the operation of the step (S800). For example, by the operation of the above step (S800), the fourth graphic object (304) displayed on the AR navigation screen of FIGS. 6 and 7 may be removed. In addition, by the operation of the above step (S800), the interval length of the driving error displayed on the error view area 340 included in the AR navigation screen of FIGS. 6 and 7 may be reduced compared to before the operation of the step (S800). For example, by the operation of the above step (S800), the interval length of the driving error displayed on the error view area (340) of the AR navigation screen of FIGS. 6 and 7 may be changed to 0 m.

[0212] FIG. 9 is a detailed flowchart of a process for eliminating an over-the-track error (OTE) of an agricultural vehicle by providing a GUI for AB driving reference line calibration on a display according to various embodiments of the present application.

[0213] For example, the process of the exemplary embodiment of FIG. 9 may be performed by a computing device, such as the processor of FIG. 2, executing a program to drive the AR system (20) of FIG. 1 or an agricultural vehicle. The process of FIG. 9 illustrated in FIG. 3 may include one or more steps, operations, functions, or actions, as exemplified by blocks (S910 to S930). Meanwhile, the schematic operations illustrated in FIG. 9 are provided only as examples, and some of the operations may be optional, may be combined into fewer operations, or may be expanded into additional operations, without departing from the essence of the disclosed embodiment.

[0214] Referring to FIG. 9, the step (S800) may begin with a step (S810) in which the AR system (20) acquires a map image including at least one map element based on GPS coordinates. The AR system (20) may acquire a map including at least one map element from a network computing device (30).

[0215] In the above step (S810), the map image may be displayed on a second display installed in the agricultural vehicle or a user terminal (100) communicating with the agricultural vehicle. The map image displayed on the second display or the user terminal (100) may be a map image that represents a geographic range that is the same as or different from the map image displayed through the AR navigation screen on the first display device.

[0216] For example, in the above step (S810), the map image may represent a wider geographic area than the map image displayed on the map view area (310).

[0217] In some embodiments, the map in step (S810) may be a real map image captured by a satellite (40). The present application uses a map image captured by a satellite (40), thereby enabling a user to intuitively recognize the location of an agricultural vehicle without feeling a discrepancy between the actual location of the agricultural vehicle and the location displayed on the map image, in contrast to cases where a virtual map image or a simplified map image using computer graphics is used.

[0218] According to one embodiment of the present application, satellite images may be acquired from an external network computing device (e.g., Google Maps, Bing Maps, Apple Maps, or NASA Worldview). These satellite images may be constructed as a database by the network computing device (30), or may be directly acquired by the AR system (20) through a service provided by the external network computing device.

[0219] In the above step (S810), the GPS coordinates may indicate the location of the agricultural vehicle currently recognized by the AR system (20). If the GPS coordinates are acquired by a GPS module included in the agricultural vehicle, the AR system (20) may receive the corresponding GPS coordinates from the agricultural vehicle.

[0220] The above navigation module (110) can acquire the map image of step (S810) using the GPS location information acquired in step (S100). The AR system (20) can transmit the GPS coordinates acquired in step (S100) to the network computing device (30) to request a map image corresponding to the GPS coordinates (S810).

[0221] In one embodiment of the present application, the AR system (20) may receive all map images within a certain range from the network computing device (30) based on GPS coordinates in step (S810). Alternatively, in one embodiment, the AR system (20) may request and acquire map images from the network computing device (30) periodically or when the agricultural vehicle moves a certain distance. Alternatively, in one embodiment, the AR system (20) may receive map images from the network computing device (30) in real time.

[0222] In the present application, map elements may refer to elements that help users easily recognize the map image displayed in the AR system (20) and the location of agricultural vehicles, such as contour lines and major place names.

[0223] In step (S820), the AR system (20) can display an AB driving reference line on a map image on a calibration interface screen. The calibration interface screen is a GUI for performing the calibration operation of step (S800), and can include images, text, objects, modals, indicators, etc. displayed on a display (131) by an application (or computer program) running on the AR system (20).

[0224] FIG. 10 illustrates an exemplary screen of an interface for calibration according to various embodiments of the present application.

[0225] Referring to FIG. 10, the AR system (20) can display the map image (710) acquired in step (S810) and the AB driving reference line (720) on the map image (710). For example, the screen illustrated in FIG. 10 can be displayed by an application running on the AR system (20).

[0226] The map image (710) may correspond to GPS coordinates and may include a contour line (711) as at least one map element. In addition, the map image (710) may include status information (712) indicating the driving speed, distance, work area, and work time of the agricultural vehicle.

[0227] An AB driving reference line (720) may be displayed on a map image (710). The AB driving reference line (720) may refer to a path from point A to point B on the map image (710). The AB driving reference line (720) may be, for example, a straight line, but is not limited thereto. Depending on the shape of the path between point A and point B, the reference line (720) may at least partially have a curved section.

[0228] Point A on the AB driving reference line (720) is the recognized position of the agricultural vehicle currently recognized by the AR system (20), indicated by the GPS coordinates of the agricultural vehicle, and can be regarded as the current position on the map image (710).

[0229] In various embodiments of the present application, the AB driving reference line (720) may be at least a portion of a driving guidance path. For example, the AB driving reference line (720) may have the current recognized position at the time of calibration in the driving guidance path in step (S200) as point A, and any point on the path between point A and the destination point of the driving guidance path as point B. In some embodiments, the point B may be any point on a straight section of the path between point A and the destination point of the driving guidance path.

[0230] In step (S830), the AR system (20) can calibrate the error of the AB driving reference line based on user input.

[0231] In one embodiment, a user may calibrate a map image (710) by performing a touch input on a left area (731) and a right area (732) based on an AB driving reference line (720). Calibration may be a correction of a position indicated by GPS coordinates and an actual position of an agricultural vehicle. If calibration is performed in step (S830), the map image (710) and the AB driving reference line (720) displayed on the map image (710) may be updated according to the corrected position. In addition, if calibration is performed in step (S830), the display of a contour line (711) as a map element displayed on the map image (710) may also be updated according to the corrected position.

[0232] A user can fix the steering wheel of an agricultural vehicle to drive in a straight line along the AB driving reference line (720). If the user recognizes that the actual driving path of the agricultural vehicle is to the left of the AB driving reference line (720), the user can calibrate by touching the left area (731) of the AB driving reference line (720) on the map image (710) displayed on the AR system (20).

[0233] Alternatively, if the user perceives that the actual driving path of the agricultural vehicle is to the right of the AB driving reference line (720), the user can calibrate by touching the right area (732) of the AB driving reference line (720) on the map image (710) displayed on the AR system (20).

[0234] There can be an error of several centimeters, or even meters, between the position indicated by GPS coordinates and the actual position of an agricultural vehicle. For agricultural vehicles that must autonomously navigate along precise paths, such as a seed-planting tractor, even an error of 10 centimeters may be unacceptable.

[0235] Even in such cases, the GUI provided by the system (10) for providing an AR navigation screen for an agricultural vehicle according to one embodiment of the present application can provide a user with a map image (710) photographed by an actual satellite, and display an AB driving reference line (720) on the map image (710), while providing a user experience (UX) that allows detailed calibration with only a simple and intuitive touch input.

[0236] Hereinafter, specific processes for calibration provided by a system (10) that provides an AR navigation screen for an agricultural vehicle according to various embodiments of the present application will be described.

[0237] FIG. 11 is a detailed flowchart of a calibration step (S830) according to various embodiments of the present application, and FIG. 12 illustrates an exemplary screen that displays a reference line movement icon overlapping the AB driving reference line.

[0238] Referring to FIG. 11, step (S830) may start with step (S831) of displaying a reference line movement icon on the calibration interface screen by overlapping the AB driving reference line.

[0239] For example, when a user inputs a calibration button (713) that triggers calibration displayed on a map image (710), a modal (730) for calibration may be displayed overlaid on the map image (710) as shown in FIG. 12. The modal (730) may display an AB driving reference line (720) and a reference line movement icon (740). In one embodiment, the modal (730) may be a pop-up window.

[0240] In the step of displaying the AB driving reference line (720) on the map (710) based on the GPS coordinates on the calibration interface screen provided by the second display or the user terminal, when a calibration button (713) that triggers calibration displayed on the map (710) is pressed, a modal (730) for calibration is displayed by overlapping it on the map (710). The modal (730) for calibration displays the AB driving reference line (720), the reference line movement icon (740), and the AB driving reference line movement indicator (750), and the left area (731) and the right area (732) are located on the modal (730).

[0241] In the above step (S831), the AR system (20) receives a user input for the interface screen for calibration, and in the above step (S832), the AR system (20) can display the movement of the reference line movement icon on the output device (141) of the second display unit (140) in response to the user input for the interface screen for calibration.

[0242] In various embodiments of the present application, the degree of calibration may be based on the number of touches or the duration of touch input by the user. For example, if the user touches the left area (731) of the modal (730) once, the agricultural vehicle may be calibrated to be located 1 cm to the left of the AB driving reference line (720). In another example, if the user touches and holds the left area (731) for 1 second, the agricultural vehicle may be calibrated to be located 1 cm to the left of the AB driving reference line (720).

[0243] In one embodiment of the present application, the AR system (20) may further display arrow objects to intuitively recognize the left area (731) and the right area (732). In another embodiment, the AR system (20) may also display separate button objects whose touch areas are visually divided, unlike the left area (731) and the right area (732).

[0244] In some embodiments, the vertical length of the set area (731, 732) may correspond to the vertical length of the AB driving reference line (720) on the display screen. For example, the height of the area (731, 732) may be the length of the AB driving reference line (720). The horizontal length of the set area (731, 732) may be the distance between the AB driving reference line (720) and the screen.

[0245] In various embodiments of the present application, in response to an input in the left area (731) or the right area (732) on the modal (730), the AR system (20) can move and display a baseline movement icon (740).

[0246] The user can intuitively accurately match the actual location of the agricultural vehicle to the map image (710) by moving the baseline movement icon (740) and comparing it with the AB driving baseline (720).

[0247] When the user completes calibration, the AR system (20) can reset the position reference of the agricultural vehicle based on the reference line movement icon (740) moved on the output device (141). Completion of calibration can be identified by receiving an input of the completion button (734) displayed on the modal (730).

[0248] In various embodiments of the present application, a movement distance numeric field (733) may be displayed in the modal (730). The movement distance numeric field (733) may numerically express the movement distance of the baseline movement icon (740) moved by the user's input.

[0249] Meanwhile, in one embodiment of the present application, the user can move the baseline movement icon (740) by directly entering a numerical value into the movement distance numerical field (733), rather than inputting the numerical value into the left area (731) or right area (732) of the modal (730). Meanwhile, the AR system (20) can initialize the movement of the baseline movement icon (740) when it detects a touch input to the 'RESET' button positioned adjacent to the movement distance numerical field (733).

[0250] FIG. 13 illustrates direct input of the movement distance of the reference line movement icon (740) in the calibration interface screen according to various embodiments of the present application.

[0251] Referring to FIG. 13, when a user receives a touch input for a movement distance numeric field (733) as a text box, the AR system (20) may further display a numeric pad (735) for numeric input at the bottom of the calibration interface screen. The user may also directly input the movement distance of the baseline movement icon (740) through input to the numeric pad (735).

[0252] FIG. 14 is a detailed flowchart of a calibration step (S830) according to various embodiments of the present application.

[0253] The detailed processes of FIG. 14 are similar to the detailed processes of FIG. 11, and may further include steps (S833 to S835) additionally performed between the step (S832) of receiving user input for the calibration interface screen illustrated in FIG. 11 and the step (S836) of displaying movement of the baseline movement icon on the calibration interface screen in response to the user input.

[0254] Referring to FIG. 14, the process (500) may start at step (S832) where the AR system (20) receives a user input for the interface screen for calibration, and then at step (S833) where it determines whether the movement of the previous baseline movement icon is within a preset movement range.

[0255] Here, the preset movement section may mean a distance that is difficult for the user to perceive due to the map scale limit when the movement distance of the baseline movement icon (740) is minute in units of cm. For example, the preset movement section may mean within 100 cm based on the AB driving baseline (720).

[0256] However, if the movement of the previous baseline movement icon in step (S833) is determined to be outside the preset movement range, the process may proceed to step (S834). In other words, if the movement of the previous baseline movement icon in step (S833) is determined to be outside the preset movement range, the process may proceed to step (S834).

[0257] In the case where the movement of the previous baseline movement icon in step (S833) is judged to be outside the preset movement section, the display of the moved baseline movement icon (740) can be recognized by the user by comparing it with the AB driving baseline (720) displayed on the map image (710).

[0258] In step (S833), if the movement of the previous baseline movement icon is determined to be within the preset movement range, the process (500) may be terminated and proceed to step (S834).

[0259] In step (S834), if the baseline movement icon (740) is moved within a preset movement range, for example, 100 cm, the AR system (20) may further display an AB driving baseline movement indicator (750). In one embodiment of the present application, the AB driving baseline movement indicator (750) may be displayed in a portion of the modal (730).

[0260] The AB driving reference line movement indicator (750) may include an auxiliary AB driving reference line (751) and an auxiliary movement line icon (752). The auxiliary AB driving reference line (751) is displayed at the center of the AB driving reference line movement indicator (750) and, like the AB driving reference line (720), may serve as a reference for the movement of the auxiliary movement line icon (752).

[0261] The auxiliary movement line icon (752) can move in response to a touch input to the left area (731) and / or the right area (732) that causes movement of the baseline movement icon (740).

[0262] According to various embodiments of the present application, the GUI for AB driving reference line calibration provided by the system (10) providing an AR navigation screen for an agricultural vehicle enables fine adjustment in units of cm by providing an AB driving reference line movement indicator (750).

[0263] In various embodiments of the present application, the AB driving reference line movement indicator (750) may provide different increments depending on the movement distance of the auxiliary movement line icon (752) or the reference line movement icon (740). For example, in the example of FIG. 8D, the AB driving reference line movement indicator (750) may provide an increment of 10 cm when the auxiliary movement line icon (752) moves to either end of the calibration interface screen.

[0264] FIG. 15 illustrates an interface screen for calibration in which an auxiliary movement line icon (752) and an auxiliary AB driving reference line (751) are displayed according to various embodiments of the present application, and FIG. 16 illustrates direct input of a movement distance of an auxiliary movement line icon (752) in the interface screen for calibration of FIG. 15.

[0265] FIG. 15 illustrates an example in which the auxiliary movement line icon (752) is moved 3 cm to the right relative to the auxiliary AB driving reference line (751). As another example, in the example of FIG. 16, the AB driving reference line movement indicator (750) can provide an increment of 100 cm movement when the auxiliary movement line icon (752) is moved to either end of the calibration interface screen. FIG. 16 illustrates an example in which the auxiliary movement line icon (752) is moved 50 cm to the right relative to the auxiliary AB driving reference line (751).

[0266] In one embodiment, the AR system (20) can be linked to multiple agricultural vehicles. In other words, the AR system (20) can establish communication with multiple agricultural vehicles. The AR system (20) can acquire map data from the agricultural vehicle with the best communication connection quality and lowest latency among the multiple agricultural vehicles.

[0267] In this way, according to various embodiments of the present application, the GUI for AB driving reference line calibration provided by the system (10) providing an AR navigation screen for agricultural vehicles can provide a smoother user experience by providing an alternative technology in cases where smooth acquisition of map data photographed by a satellite is difficult.

[0268] The above navigation module (110) can generate an AR navigation screen reflecting the calibration result when the GPS point and the map displayed on the display are accurately matched in the autonomous driving environment of the agricultural vehicle according to the calibration operation (S800). Then, the navigation module (110) can correct the driving error information of the agricultural vehicle according to the calibration result and generate the AR navigation screen showing the corrected driving error of the agricultural vehicle (S700). In the AR navigation screen showing the corrected driving error, the AR graphic object (304) indicating the corrected driving error is relatively reduced compared to before calibration. For example, if the corrected driving error is 0 m, the AR graphic object (304) is removed from the AR navigation screen showing the corrected driving error. Therefore, the calibration operation (S800) can be treated as an operation of removing the AR graphic object (304) from the existing AR navigation screen.

[0269]

[0270]

[0271] It will be apparent to those skilled in the art that the agricultural vehicle (20) may include other components. For example, the agricultural vehicle (20) may include other hardware elements necessary for the operations described herein, including input devices for data entry and output devices for printing or other data display.

[0272] When implementing an embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), etc. configured to perform the present invention may be provided in the processor of the present invention.

[0273] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable storage medium through various means. It should be understood that the program storage devices that can be used to describe a storage device including executable computer code for performing various methods of the present invention do not include transient objects such as carrier waves or signals. The computer-readable storage medium includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0274] The embodiments described above are combinations of components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented without being combined with other components or features. Furthermore, it is also possible to form an embodiment of the present invention by combining some components and / or features. The order of operations described in the embodiments of the invention may be changed. Some components or features of one embodiment may be included in another embodiment or may be replaced with corresponding components or features of another embodiment. It is self-evident that claims that are not explicitly cited in the scope of the patent may be combined to form an embodiment or may be incorporated as a new claim through a post-application amendment.

[0275] It will be apparent to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential characteristics of the present invention. Therefore, the above embodiments should be considered illustrative rather than restrictive in all respects. The scope of the present invention should be determined by a reasonable interpretation of the appended claims and all possible variations within the scope of equivalents of the present invention.

[0276] The embodiments of the present application are expected to have high usability in the agricultural vehicle industry, as they can effectively convey various information related to agricultural vehicles in a visual aspect through augmented reality technology.

Claims

1. A method for providing an augmented reality (AR) navigation screen, performed by a system providing an augmented reality (AR) navigation screen installed in an agricultural vehicle, A step of obtaining GPS location information indicating the GPS location of the agricultural vehicle from a GPS module; A step of obtaining navigation information of the agricultural vehicle - the navigation information includes driving guidance route information describing a driving guidance route to be driven by the agricultural vehicle; A step of acquiring a driving environment image of the driving environment in which the agricultural vehicle is driving from a camera module; A step of calculating driving-related information of an agricultural vehicle based on GPS location information of the agricultural vehicle acquired while the agricultural vehicle is driving; A step of obtaining the difference between the driving guidance path and the actual driving path of the agricultural vehicle as driving error information of the agricultural vehicle; A step of generating at least one AR graphic object that visually expresses at least one of the above navigation information, driving-related information of the agricultural vehicle, and driving error information, and generating an AR navigation screen in which the AR graphic object is placed on the driving environment image; comprising; method.

2. In the first paragraph, the step of generating the AR navigation screen comprises: A step of identifying a location of a graphic object representing at least two pieces of information from among the above navigation information, driving-related information, and driving error information - the location of the AR graphic object is a location on the driving environment image; A step of defining a shape space of each graphic object based on the identified location, and generating a shape of the AR graphic object based on the defined shape space; and A method characterized by comprising the step of overlaying the shape of the AR graphic object on a location on the identified driving environment image to generate an AR navigation screen in which the shape of the AR graphic object is expressed on the driving environment image. method.

3. In paragraph 2, The above AR graphic object includes a first graphic object that visually represents a driving guidance path, The step of identifying the location of the AR graphic object on the driving environment image is: A step of identifying the location of the first graphic object based on GPS location information of the agricultural vehicle, map information for a geographical area including the GPS location of the agricultural vehicle, driving direction information of the agricultural vehicle, and parameters of the camera module; The step of identifying the location of the first graphic object is: A step of mapping a two-dimensional coordinate system for the above driving environment image and a three-dimensional coordinate system for a virtual three-dimensional world; A step of calculating a 3D location of the driving guidance route in the virtual 3D world based on the map information and the driving guidance route information; and A step of changing the three-dimensional position of the driving guidance path to the two-dimensional position of the driving guidance path through a mapping relationship between the two-dimensional coordinate system and the three-dimensional coordinate system, and identifying the changed two-dimensional position as the position of the first graphic object; method.

4. In the third paragraph, the step of mapping a two-dimensional coordinate system and a three-dimensional coordinate system for a virtual three-dimensional world in the driving environment image is, A step of mapping the two-dimensional coordinates of the driving environment image and the virtual three-dimensional world coordinates by setting the internal parameters of the camera module, The intrinsic parameters of the camera module are characterized in that they include at least one of a principal point, a focal length, and a distortion coefficient. method.

5. In paragraph 3, The above driving-related information includes one or more of driving mode information, a driving route driven along the driving guidance route, a driving distance according to the driving route, a driving time, real-time driving speed of the agricultural vehicle at the current location at the time of generating the AR navigation screen, a driving direction angle of the agricultural vehicle, and work result area information. The above AR graphic object further includes at least one graphic object among a second graphic object that visually represents a work result area according to driving of the agricultural vehicle, a third graphic object that visually represents a driving direction, and a fourth graphic object that visually represents the driving error information. The step of identifying the location of the AR graphic object on the driving environment image is: When the AR graphic object includes the second graphic object, a step of identifying the location of the second graphic object based on the location of the identified first graphic object, driving path information, driving distance information, and unit work result area information per unit driving distance stored in advance. When the AR graphic object includes the third graphic object, a step of identifying the position of the third graphic object based on the position and driving direction information of the identified first graphic object; and If the AR graphic object includes the fourth graphic object, a step of identifying the position of the fourth graphic object based on the position of the identified first graphic object and the driving error information is included. The positions of the second graphic object, the third graphic object, and the fourth graphic object are characterized in that they are positions on a two-dimensional coordinate system for the driving environment image. method.

6. In paragraph 1, The steps for creating the above AR navigation screen are: A step of generating a map view area showing GPS location information of the agricultural vehicle, a map image, driving direction information of the agricultural vehicle, work result area information, and a surrounding area of ​​the agricultural vehicle; A step of generating an error view area that describes the current driving error of the agricultural machine at the time of generating the AR navigation screen based on the driving error information; and In order to generate the AR navigation screen in which at least one of the map view area and the error view area appears, further comprising a step of overlaying at least one of the map view area and the error view area on the driving environment image. method.

7. In the first paragraph, the step of generating the AR navigation screen comprises: A step of displaying the driving distance information, the area of ​​the work result area, and the driving time information of the agricultural vehicle in the generated driving-related information in the first sub-area of ​​the navigation screen; A step of displaying driving mode information of the agricultural vehicle in a second sub-area of ​​the navigation screen; and Further comprising a step of displaying the contents of the driving guidance message in the acquired navigation information in a sub-area of ​​the navigation screen. method.

8. In the first paragraph, the method, In order to eliminate the driving error of the agricultural vehicle, the step of displaying an AB driving reference line based on the GPS position of the agricultural vehicle on a calibration interface screen including a map image on which the agricultural vehicle appears, and calibrating the GPS coordinates of the agricultural vehicle to the actual coordinates of the agricultural vehicle using the AB driving reference line is further included. The above AB driving reference line is at least a part of the driving guidance route in the above navigation information, In order to indicate the driving error of the agricultural vehicle corrected according to the calibration result on the AR navigation screen, the AR graphic object indicating the driving error of the agricultural vehicle corrected according to the calibration result on the AR navigation screen is characterized in that it is relatively reduced compared to before calibration. method.

9. In the 8th paragraph, the calibration step: A step of displaying a reference line movement icon on the calibration interface screen by overlapping the AB driving reference line; A step of moving the reference line movement icon displayed on the calibration interface screen in response to a user input on the calibration interface screen; and Including a step of additionally displaying an AB driving baseline movement indicator on the calibration interface screen when the reference line movement icon displayed on the calibration interface screen moves within a preset movement range in response to a user input on the calibration interface screen, The above AB driving reference line movement indicator is located in the outer area of ​​the left area and the right area of ​​the interface screen for calibration, and when the movement of the reference line movement icon is performed outside the preset movement section, the display of the AB driving reference line movement indicator is deactivated. The user input for the calibration interface screen is a user's touch input for a preset area based on a touch interface built into the calibration interface screen, and a left area or a right area based on the AB driving reference line on the calibration interface screen is an area set to receive the user's touch input, and when the user's touch input is made in the left area, the reference line movement icon is moved to the left by a preset distance, and when the user's touch input is made in the right area, the reference line movement icon is moved to the right by a preset distance. The above AB driving reference line movement indicator includes an auxiliary AB driving reference line and an auxiliary moving line icon that serve as a reference for the movement of an auxiliary moving line icon, and the auxiliary AB driving reference line and the auxiliary moving line icon are displayed together on the AB driving reference line and the calibration interface screen. The auxiliary AB driving reference line is displayed in the center of the AB driving reference line movement indicator, and the auxiliary moving line icon is characterized in that it moves in response to a touch input to the left area or the right area that causes movement of the reference line movement icon in the AB driving reference line movement indicator. method.

10. In paragraph 9, In the step of displaying the AB driving reference line based on the GPS location of the agricultural vehicle on the calibration interface screen including the map image where the agricultural vehicle appears, when the calibration button that triggers the calibration displayed on the map is input, a modal for calibration is displayed by overlaying it on the map. The modal for the above calibration displays the AB driving reference line, the reference line movement icon, and the AB driving reference line movement indicator. The left and right areas are characterized in that they are located on the modal. method.

11. A computer-readable recording medium recording a program for performing a method for generating an augmented reality (AR) navigation screen according to any one of claims 1 to 10.

12. In a navigation module installed in an agricultural vehicle and generating an augmented reality (AR) navigation screen, the navigation module stores a processor and a memory including commands, and the commands, when executed by the processor, A step of obtaining GPS location information indicating the GPS location of the agricultural vehicle from a GPS module; A step of obtaining navigation information of the agricultural vehicle - the navigation information includes driving guidance route information describing a driving guidance route to be driven by the agricultural vehicle; A step of acquiring a driving environment image of the driving environment in which the agricultural vehicle is driving from a camera module; A step of calculating driving-related information of an agricultural vehicle based on GPS location information of the agricultural vehicle acquired while the agricultural vehicle is driving; A step of obtaining the difference between the driving guidance path and the actual driving path of the agricultural vehicle as driving error information of the agricultural vehicle; It is characterized by including a command for generating at least one AR graphic object that visually expresses at least one of the above navigation information, driving-related information of the agricultural vehicle, and driving error information, and performing a step of generating an AR navigation screen in which the AR graphic object is placed on the driving environment image. Navigation module.

Images