Vehicle exterior image auxiliary driving method, device and equipment, medium and forklift

Abstract

The invention relates to an outside-vehicle image auxiliary driving method, device and equipment, a medium and a forklift, and the method comprises the steps that current image data collected by all cameras are received, and the current image data collected by different cameras are image data collected from different angles outside a vehicle owner; splicing the current image data to obtain a panoramic image outside the vehicle owner; and sending the panoramic image to a visual display, so that the visual display provides the panoramic image for a driver in the vehicle owner. According to the method, the driving operation of the driver is assisted through the panoramic image outside the vehicle, the problem that the driver has a view blind area outside the vehicle is greatly solved, a wider view angle is provided, the driver can conveniently observe surrounding conditions in real time, and the possibility of collision in the driving process is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of forklift transportation, and in particular to an off-vehicle image-assisted driving method, device, equipment, medium and forklift. Background Art

[0002] In domestic natural gas stations, when a forklift is transporting large volumes of cargo, the cargo may block the driver's line of sight, making it difficult for the driver to observe the surrounding environment and the path, and unable to detect obstacles in front in time, resulting in the operator being unable to accurately judge the relative position of the cargo and the surrounding environment, thereby increasing the risk of collision between the forklift and pedestrians, other vehicles or fixed objects. Obstructed vision may also cause the driver to be unable to correctly identify the path, thereby choosing the wrong driving route, increasing the risk of accidents. Summary of the invention

[0003] In order to assist the driver in driving safely, the present application provides an off-vehicle image assisted driving method, device, equipment, medium and forklift.

[0004] In a first aspect, the present application provides an external image assisted driving method, comprising:

[0005] Receiving current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body;

[0006] splicing the current image data to obtain a panoramic image outside the vehicle body;

[0007] The panoramic image is sent to a visualization display so that the visualization display provides the panoramic image to a driver in the vehicle body.

[0008] The beneficial effects of the present application are as follows: the driver's driving operation is assisted by the panoramic image outside the vehicle, which greatly solves the problem of the driver's blind spot outside the vehicle, provides a wider viewing angle, facilitates the driver to observe the surrounding situation in real time, and reduces the possibility of collision during driving.

[0009] Further, after obtaining the panoramic image outside the vehicle body, the method further includes:

[0010] Identify multiple target elements in the panoramic image, where the target elements are vehicles, pedestrians or obstacles;

[0011] For each target element, based on the attribute information of the target element, determining the hazard level and marking content corresponding to the target element, the attribute information including the vehicle type, the age of the pedestrian and the obstacle value, the hazard level being low risk, medium risk or high risk, and the marking content including the speed, size and the distance between the target element and the vehicle body;

[0012] Based on each of the hazard levels, obtaining a marking icon corresponding to each of the target elements, each of the marking icons representing a logo shape, color or pattern of the corresponding target element;

[0013] The panoramic image is marked based on each of the marking contents and each of the marking icons, and the marked panoramic image is sent to the visual display.

[0014] The beneficial effect of adopting the above further scheme is that through the comprehensive use of target elements and their attribute information, accurate identification of target elements in panoramic images, assessment of hazard levels, and generation and display of marking content can be achieved, thereby providing the driver with real-time and accurate driving assistance information.

[0015] Further, after determining the hazard level and the marking content corresponding to each of the target elements, the method further includes:

[0016] Based on each of the attribute information and each of the marking contents, determining a safe area and an obstacle area in the panoramic image, wherein the safe area is an area where a forklift can travel safely, and the obstacle area is an area that needs to be avoided;

[0017] Based on the turning radius and driving speed of the vehicle body, obtaining an optimal moving path of the vehicle body in the safety area and an optimal driving speed corresponding to the optimal moving path;

[0018] Based on the optimal moving path and the optimal driving speed, path guidance information is generated, and the path guidance information is sent to an in-vehicle guidance device.

[0019] The beneficial effects of adopting the above further solution are: by determining the safe area and the obstacle area, the obstacle information can be provided to the driver in real time, helping the driver to avoid collision with obstacles, and planning the optimal moving path not only reduces the uncertainty during driving, but also reduces the risk of traffic accidents caused by improper path selection. By determining the optimal driving speed, it helps the driver to maintain a reasonable speed and avoid safety problems caused by too fast or too slow speed.

[0020] Furthermore, the current image data are stitched together to obtain a panoramic image outside the vehicle body, including:

[0021] If there is a target area blocked by the cargo in the area captured by the camera, determining whether there is auxiliary image data corresponding to the target area captured by the camera of another vehicle body;

[0022] If the auxiliary image data exists, the auxiliary image data is subjected to image registration processing with each of the current image data, and image restoration processing is performed on the image data after the image registration processing to obtain the panoramic image;

[0023] If the auxiliary image data does not exist, determining whether the target area is a fixed area, and whether there is no object whose overall position changes in the fixed area;

[0024] If the target area is a fixed area, historical image data corresponding to the target area is acquired, and image stitching processing is performed based on the historical image data and each of the current image data to obtain the panoramic image.

[0025] The beneficial effect of adopting the above further solution is that by searching for auxiliary image data and processing the auxiliary image data and current image data, the area blocked by goods etc. can be filled, making the image more complete and providing more comprehensive visual information. Even when the camera field of view is partially blocked, a complete panoramic picture can be obtained, ensuring the real-time and effectiveness of assisted driving.

[0026] In a second aspect, the present application provides an external image driving assistance device, comprising:

[0027] A data receiving module is used to receive current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body;

[0028] A panoramic stitching module is used to stitch the current image data to obtain a panoramic image outside the vehicle body;

[0029] The sending and displaying module is used to send the panoramic image to a visual display so that the visual display provides the panoramic image to the driver in the vehicle body.

[0030] In a third aspect, the present application provides an electronic device, including a processor and a memory, wherein the processor is coupled to the memory;

[0031] The processor is used to execute the computer program stored in the memory so that the electronic device performs the method as described in any one of the first aspects.

[0032] In a fourth aspect, the present application provides a computer-readable storage medium, comprising a computer program or instructions, which, when executed on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0033] In a fifth aspect, the present application provides a forklift using the method described in any one of the first aspects, comprising a vehicle body, on which are mounted a plurality of cameras, a visual display, a data transmission line and a control system; different cameras are used to collect current image data at different angles outside the vehicle body; the data transmission line is used to transmit the current image data collected by the plurality of cameras to the control system; the control system is used to perform splicing processing on the received current image data, and transmit the spliced ​​data to the visual display; the visual display is installed inside the vehicle body, and the visual display is used to display auxiliary images outside the vehicle to the driver.

[0034] Furthermore, a vehicle door is installed on the vehicle body, and a distance sensor and an angle sensor are provided on the vehicle door. The distance sensor is used to detect the distance information between the vehicle body and an obstacle, and the angle sensor is used to detect the moving angle information of the vehicle body. The data transmission line is also used to transmit the distance information and the moving angle information to the control system.

[0035] Furthermore, the cameras are all explosion-proof cameras, and there are three explosion-proof cameras, and the corresponding viewing angle of each explosion-proof camera is 120 degrees. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of the method for assisting driving with external vehicle image in an embodiment of the present application;

[0037] Figure 2 This is a structural block diagram of the vehicle exterior image auxiliary driving device in an embodiment of the present application;

[0038] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application is further described in detail below in conjunction with the accompanying drawings.

[0040] The embodiment of the present application provides an off-vehicle image assisted driving method, which can be executed by a device, which can be a server or a terminal device, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a tablet computer, a desktop computer, etc., but is not limited thereto.

[0041] like Figure 1 As shown, a method for assisting driving with an external image is described, with an electronic device as the execution subject. The main process of the method is described as follows (steps S101 to S103):

[0042] Step S101: receiving current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body.

[0043] Multiple cameras are installed on the vehicle body, and the cameras can be explosion-proof cameras. The shooting area and shooting direction of each camera are different. In this embodiment, the number of cameras can be three, and the corresponding viewing angle of each camera can be 120 degrees. The three cameras are respectively connected to the left, right and rear rearview mirrors of the vehicle body.

[0044] Each camera is respectively connected to the electronic device for communication and is used to transmit the collected current image data to the electronic device.

[0045] Step S102: stitching the current image data to obtain a panoramic image outside the vehicle body.

[0046] Since the shapes and sizes of goods transported by forklifts are different, when the size of the goods is large, there is a possibility that the goods will block the camera shooting. If there is no target area blocked by the goods in the area captured by the camera, the step S102 specifically includes:

[0047] The current image data of each camera is preprocessed, including denoising, grayscale, edge detection and other steps to improve the image quality; the feature points in each current image data are detected using a feature point matching algorithm (such as SIFT, SURF, ORB, etc.), and the feature descriptors are calculated; the images are matched according to the feature descriptors to determine the relative position relationship between the images; the current image data are transformed into the same coordinate system using an image transformation algorithm (such as affine transformation, perspective transformation) to form a preliminary stitched image; the stitched image is fused to eliminate the stitching gaps and overlapping areas between the images to obtain the panoramic image.

[0048] Step S103: sending the panoramic image to a visual display, so that the visual display provides the panoramic image to the driver in the vehicle body.

[0049] Since panoramic images usually contain a large amount of image data, in order to efficiently transmit and reduce bandwidth usage, the original image data can be compressed and encoded through image compression algorithms. Image compression algorithms can include JPEG, PNG, etc., and for video streams, efficient video encoding standards such as H.264 and H.265 can be used.

[0050] In this embodiment, to ensure the real-time and reliability of the data, the panoramic image data needs to be transmitted to the visual display through a preset communication protocol, such as UDP (User Datagram Protocol) for real-time video streaming transmission, or TCP (Transmission Control Protocol) for data transmission scenarios that require higher reliability.

[0051] In this embodiment, the visual display may be a high-resolution display, which can provide more delicate and clear images, thereby enhancing the driver's visual experience. The visual display may also have interactive functions, such as zooming, rotating, and panning, so that the driver can adjust the viewing angle or view details as needed. In addition, the visual display may also be integrated with a touch screen or other input devices to achieve a more intuitive interactive experience.

[0052] The driver's driving operations are assisted by panoramic images outside the car, which greatly solves the problem of the driver's blind spots outside the car. It provides a wider perspective, makes it easier for the driver to observe the surrounding situation in real time, and reduces the possibility of collision during driving.

[0053] In this embodiment, after step S102, the following steps are further included:

[0054] Identify multiple target elements in the panoramic image, where the target elements are vehicles, pedestrians or obstacles;

[0055] For each target element, based on the attribute information of the target element, determining the hazard level and marking content corresponding to the target element, the attribute information including the vehicle type, the age of the pedestrian and the obstacle value, the hazard level being low risk, medium risk or high risk, and the marking content including the speed, size and the distance between the target element and the vehicle body;

[0056] Based on each of the hazard levels, obtaining a marking icon corresponding to each of the target elements, each of the marking icons representing a logo shape, color or pattern of the corresponding target element;

[0057] The panoramic image is marked based on each of the marking contents and each of the marking icons, and the marked panoramic image is sent to the visual display.

[0058] In this embodiment, the target detection algorithm in deep learning (such as YOLO, SSD, Faster R-CNN, etc.) can be used to analyze the panoramic image in real time to identify target elements such as vehicles, pedestrians and obstacles. The target detection algorithm can learn the characteristics of the target elements by training a large amount of image data, and accurately detect the corresponding position and attribute information of the target elements in the new image.

[0059] Vehicle types may include utility vehicles, sedans, SUVs, etc. For each identified target element, the hazard level corresponding to the target element is determined based on different combinations of attribute information. At the same time, the corresponding marking content is also generated based on information such as the target element's speed, size, and distance from the vehicle body.

[0060] According to the hazard level of the target element, a corresponding marking icon is selected from a preset icon library. The marking icon is used to identify the target element on the display interface of the visual display. The marking icon usually has a specific shape, color or pattern so that the driver can intuitively identify and understand its meaning.

[0061] Through image synthesis technologies such as image overlay and transparency adjustment, the selected mark icon is superimposed on the corresponding target element position in the panoramic image to form a marked panoramic image. The mark content can be updated and dynamically displayed in real time according to the panoramic image to adapt to changes in the vehicle during driving.

[0062] By comprehensively using the target elements and their attribute information, it is possible to accurately identify the target elements in the panoramic image, evaluate the hazard level, and generate and display the marking content, thereby providing the driver with real-time and accurate driving assistance information.

[0063] In this embodiment, after determining the hazard level and the marking content corresponding to each target element, the following processing is also included:

[0064] Based on each of the attribute information and each of the marking contents, determining a safe area and an obstacle area in the panoramic image, wherein the safe area is an area where a forklift can travel safely, and the obstacle area is an area that needs to be avoided;

[0065] Based on the turning radius and driving speed of the vehicle body, obtaining an optimal moving path of the vehicle body in the safety area and an optimal driving speed corresponding to the optimal moving path;

[0066] Based on the optimal moving path and the optimal driving speed, path guidance information is generated, and the path guidance information is sent to an in-vehicle guidance device.

[0067] During the driving process of the forklift, the target elements may undergo dynamic changes. Dynamic changes mainly refer to changes in spatial position, speed, direction, etc. When dividing areas, it is necessary to update the dynamic changes of the target elements in real time or near real time, and adjust the division of areas accordingly.

[0068] Based on statistical models, machine learning algorithms or deep learning networks, the travel paths corresponding to each target element are predicted according to the attribute information and tag content corresponding to each target element. In this embodiment, the possible travel trajectory and acceleration / deceleration behavior of the vehicle can be predicted according to the type and speed of the vehicle; the possible walking path and dwell time of the pedestrian can be predicted according to the position and speed of the pedestrian.

[0069] After predicting the path of the target element, the division of the area can be dynamically adjusted according to the obtained path. For example, if it is predicted that a large number of pedestrians or vehicles will gather in a certain area, the area can be divided into a "high-risk area", that is, an obstacle area; if it is predicted that the obstacles in a certain area will change (such as the movement of construction vehicles), and the obstacles will not affect the driving of forklifts after the change, the area can be divided into a "temporary obstacle area", that is, a safe area; if it is predicted that there will never be obstacles in a certain area, the area can be divided into a "no-risk area", that is, a safe area.

[0070] Based on the turning radius and driving speed of the vehicle, the path planning algorithm is used to plan the optimal moving path for the vehicle in the safe area. The path planning algorithm can consider multiple factors, such as path length, path smoothness, obstacle avoidance ability, and kinematic constraints of the vehicle. The kinematic constraints of the vehicle include turning radius, driving speed, etc. The path planning algorithm can include A* algorithm, Dijkstra algorithm, genetic algorithm, etc.

[0071] After obtaining the optimal moving path, the speed planning algorithm is used to determine the optimal driving speed based on factors such as the curvature of the optimal moving path and the density of target elements. The speed planning algorithm can include rule-based methods, optimization-based methods, etc. When determining the speed, it is necessary to consider the dynamic constraints of the vehicle body, such as maximum acceleration, maximum deceleration, etc.

[0072] Combined with map data, vehicle location information and other information, the optimal moving path and optimal driving speed are converted into path guidance information, which may include direction instructions, speed prompts, etc. Utilizing in-vehicle guidance equipment (such as display screens, voice prompts, etc.), the path guidance information is conveyed to the driver in real time, providing the driver with intuitive and clear driving assistance information, further improving the safety and convenience during driving.

[0073] By determining the safe area and the obstacle area, it can provide the driver with obstacle information in real time, helping the driver to avoid collisions with obstacles. Planning the optimal moving path not only reduces the uncertainty during driving, but also reduces the risk of traffic accidents caused by improper path selection. Determining the optimal driving speed based on factors such as the curvature of the optimal moving path and the density of target elements helps the driver maintain a reasonable speed and avoid safety problems caused by speeding too fast or too slow.

[0074] In this embodiment, step S102 specifically includes the following processing:

[0075] If there is a target area blocked by the cargo in the area captured by the camera, determining whether there is auxiliary image data corresponding to the target area captured by the camera of another vehicle body;

[0076] If the auxiliary image data exists, the auxiliary image data is subjected to image registration processing with each of the current image data, and image restoration processing is performed on the image data after the image registration processing to obtain the panoramic image;

[0077] If the auxiliary image data does not exist, determining whether the target area is a fixed area, and whether there is no object whose overall position changes in the fixed area;

[0078] If the target area is a fixed area, historical image data corresponding to the target area is acquired, and image stitching processing is performed based on the historical image data and each of the current image data to obtain the panoramic image.

[0079] When the size of the goods is large, there is a possibility that the goods will block the camera. The blocked area in the area captured by the camera can be detected through image processing algorithms such as edge detection and contour recognition.

[0080] In this embodiment, the electronic devices corresponding to the multiple forklifts are respectively connected to the server in communication, thereby establishing a communication network between the multiple forklifts, allowing each vehicle body to share camera data. When a target area blocked by cargo is detected, a request is sent to the electronic devices corresponding to other forklifts in the network to inquire whether there is auxiliary image data of the area, and the returned auxiliary image data is received and verified to ensure its quality and relevance.

[0081] The timestamp corresponding to the auxiliary image data can be the same as the timestamp corresponding to the current image data. When the electronic devices corresponding to other forklifts cannot send data with the same timestamp, the image data of the target area earlier than the current image data can also be used as auxiliary image data. The difference between the timestamps of the current image data and the auxiliary image data needs to be within a preset time range.

[0082] If the auxiliary image data exists, an image registration algorithm such as the RANSAC algorithm is used to register the auxiliary image data with the current image data, and an image restoration algorithm (such as image interpolation, image patching algorithm) is used to repair the occluded target area.

[0083] The electronic device stores the types of different areas in the current transportation scenario. The types of areas include fixed areas and non-fixed areas. Fixed areas may be equipment at fixed positions, walls, goods stacked for a long time, etc., and areas where objects with no position changes are located. Fixed areas may also be objects such as equipment with partial position changes on the surface or inside but no overall external position changes. For example, the area where a conveying equipment is located may be a fixed area. The conveying equipment includes a conveyor belt. The conveyor belt and its driving device may change position, but the overall external position of the conveying equipment is fixed.

[0084] The electronic device can obtain the current position coordinates through the position sensor on the forklift, and the current orientation through the angle sensor on the forklift. According to the position coordinates, orientation and current image data, the actual area of ​​the transportation scene corresponding to the target area can be obtained, and then the area type corresponding to the actual area of ​​the transportation scene can be obtained. If the area type is a fixed area, the target area is a fixed area; if the area type is a non-fixed area, the target area is a non-fixed area.

[0085] The historical image data of the target area is retrieved from the database, and then the historical image data is spliced ​​with the current image data through image splicing algorithms (such as feature point matching, image deformation, etc.). The spliced ​​image is post-processed to obtain a panoramic image. The post-processing includes color correction, brightness adjustment, etc., which improves the image quality.

[0086] By searching for auxiliary image data and processing the auxiliary image data with the current image data, the area blocked by goods can be filled, making the image more complete and providing more comprehensive visual information. Even when the camera's field of view is partially blocked, a complete panoramic picture can be obtained, ensuring the real-time and effectiveness of assisted driving.

[0087] Based on the same technical concept, the present application also provides an external image assist driving device, such as Figure 2 As shown, the vehicle exterior image auxiliary driving device 200 mainly includes:

[0088] The data receiving module 201 is used to receive the current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body;

[0089] A panoramic stitching module 202 is used to stitch the current image data to obtain a panoramic image outside the vehicle body;

[0090] The sending and displaying module 203 is used to send the panoramic image to a visual display, so that the visual display provides the panoramic image to the driver in the vehicle body.

[0091] Optionally, after the panorama stitching module 202, the method further includes:

[0092] A target recognition module, used to recognize multiple target elements in the panoramic image, wherein the target elements are vehicles, pedestrians or obstacles;

[0093] A first determination module is used to determine, for each target element, a hazard level and a marking content corresponding to the target element based on the attribute information of the target element, wherein the attribute information includes the vehicle type, the age of the pedestrian and the value of the obstacle, the hazard level is low risk, medium risk or high risk, and the marking content includes the speed, size and distance between the target element and the vehicle body;

[0094] An icon acquisition module is used to acquire a marking icon corresponding to each of the target elements based on each of the hazard levels, each of the marking icons representing a logo shape, color or pattern of the corresponding target element;

[0095] The image marking module is used to mark the panoramic image based on each of the marking contents and each of the marking icons, and send the marked panoramic image to the visual display.

[0096] Optionally, after the first determination module, the method further includes:

[0097] A second determination module is used to determine a safe area and an obstacle area in the panoramic image based on each of the attribute information and each of the marking contents, wherein the safe area is an area where a forklift can travel safely, and the obstacle area is an area that needs to be avoided;

[0098] A path acquisition module, used to acquire an optimal moving path of the vehicle body in the safety area and an optimal driving speed corresponding to the optimal moving path based on the turning radius and driving speed of the vehicle body;

[0099] A guidance generating module is used to generate path guidance information based on the optimal moving path and the optimal driving speed, and send the path guidance information to an in-vehicle guidance device.

[0100] Optionally, the panorama stitching module 202 includes:

[0101] A first judgment module is used to judge whether there is auxiliary image data corresponding to the target area captured by the camera of another vehicle body when there is a target area blocked by the cargo in the area captured by the camera;

[0102] A registration and restoration module, configured to perform image registration processing on the auxiliary image data and each of the current image data when the auxiliary image data exists, and perform image restoration processing on the image data after the image registration processing to obtain the panoramic image;

[0103] A second judgment module is used to judge whether the target area is a fixed area when the auxiliary image data does not exist, and there is no object with an overall position change in the fixed area;

[0104] The stitching module is used to obtain historical image data corresponding to the target area when the target area is a fixed area, and perform image stitching processing based on the historical image data and each of the current image data to obtain the panoramic image.

[0105] In one example, the module in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0106] For another example, when the modules in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0107] Various objects such as various messages / information / equipment / network elements / systems / devices / actions / operations / processes / concepts that may appear in this application are named. It can be understood that these specific names do not constitute a limitation on the relevant objects. The names assigned may change with factors such as scenarios, contexts or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from the functions and technical effects embodied / executed in the technical scheme.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] Based on the same technical concept, the present application also provides an electronic device, such as Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more of a communication component 304 , and a communication bus 305 .

[0111] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned vehicle-exterior image-assisted driving method; the memory 302 is used to store various types of data to support the operation of the electronic device 300, and these data may include, for example, instructions for any application or method used to operate on the electronic device 300, and data related to the application. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. One or more.

[0112] The I / O interface 303 provides an interface between the processor 301 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 304 is used to test the wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 304 can include: Wi-Fi components, Bluetooth components, NFC components.

[0113] The communication bus 305 may include a path to transmit information between the above components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0114] The electronic device 300 can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the off-vehicle image assisted driving method given in the above embodiment.

[0115] The electronic device 300 may include but is not limited to mobile terminals such as digital broadcast receivers, PDAs (personal digital assistants), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc., and may also be servers, etc.

[0116] Based on the same technical concept, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for assisted driving with external vehicle images are implemented.

[0117] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0118] Based on the same technical concept, the present application also provides a forklift using the above-mentioned outside-vehicle image assisted driving method, including a vehicle body, on which are installed multiple cameras, a visual display, a data transmission line and a control system; different cameras are used to collect current image data at different angles outside the vehicle body; the data transmission line is used to transmit the current image data collected by multiple cameras to the control system; the control system is used to splice the received current image data and transmit the spliced ​​data to the visual display; the visual display is installed in the vehicle body, and the visual display is used to display the auxiliary image outside the vehicle to the driver.

[0119] A door is also installed on the vehicle body, and a distance sensor and an angle sensor are provided on the door. The distance sensor is used to detect the distance information between the vehicle body and an obstacle, and the angle sensor is used to detect the moving angle information of the vehicle body. The data transmission line is also used to transmit the distance information and the moving angle information to the control system.

[0120] The cameras are all explosion-proof cameras, and three of them are provided. The corresponding viewing angle of each explosion-proof camera is 120 degrees.

[0121] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0124] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for assisting driving using external vehicle images, characterized in that: include: Receiving current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body; splicing the current image data to obtain a panoramic image outside the vehicle body; The panoramic image is sent to a visualization display so that the visualization display provides the panoramic image to a driver in the vehicle body.

2. The method for assisting driving using external vehicle images according to claim 1, characterized in that: After obtaining the panoramic image outside the vehicle body, the method further includes: Identify multiple target elements in the panoramic image, where the target elements are vehicles, pedestrians or obstacles; For each target element, based on the attribute information of the target element, determining the hazard level and marking content corresponding to the target element, the attribute information including the vehicle type, the age of the pedestrian and the obstacle value, the hazard level being low risk, medium risk or high risk, and the marking content including the speed, size and the distance between the target element and the vehicle body; Based on each of the hazard levels, obtaining a marking icon corresponding to each of the target elements, each of the marking icons representing a logo shape, color or pattern of the corresponding target element; The panoramic image is marked based on each of the marking contents and each of the marking icons, and the marked panoramic image is sent to the visual display.

3. The method for assisting driving using external vehicle images according to claim 2, characterized in that: After determining the hazard level and the marking content corresponding to each of the target elements, the method further includes: Based on each of the attribute information and each of the marking contents, determining a safe area and an obstacle area in the panoramic image, wherein the safe area is an area where a forklift can travel safely, and the obstacle area is an area that needs to be avoided; Based on the turning radius and driving speed of the vehicle body, obtaining an optimal moving path of the vehicle body in the safety area and an optimal driving speed corresponding to the optimal moving path; Based on the optimal moving path and the optimal driving speed, path guidance information is generated, and the path guidance information is sent to an in-vehicle guidance device.

4. The method for assisting driving using external vehicle images according to claim 1, characterized in that: The step of stitching the current image data to obtain a panoramic image outside the vehicle body includes: If there is a target area blocked by the cargo in the area captured by the camera, determining whether there is auxiliary image data corresponding to the target area captured by the camera of another vehicle body; If the auxiliary image data exists, the auxiliary image data is subjected to image registration processing with each of the current image data, and image restoration processing is performed on the image data after the image registration processing to obtain the panoramic image; If the auxiliary image data does not exist, determining whether the target area is a fixed area, and whether there is no object whose overall position changes in the fixed area; If the target area is a fixed area, historical image data corresponding to the target area is acquired, and image stitching processing is performed based on the historical image data and each of the current image data to obtain the panoramic image.

5. An external image driving assistance device, characterized in that: include: A data receiving module is used to receive current image data collected by each camera, wherein the current image data collected by different cameras are image data collected from different angles outside the vehicle body; A panoramic stitching module is used to stitch the current image data to obtain a panoramic image outside the vehicle body; The sending and displaying module is used to send the panoramic image to a visual display so that the visual display provides the panoramic image to the driver in the vehicle body.

6. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor is coupled to the memory; The processor is configured to execute a computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 4.

8. A forklift using the method according to any one of claims 1 to 4, characterized in that: The vehicle comprises a vehicle body, on which a plurality of cameras, a visual display, a data transmission line and a control system are installed; different cameras are used to collect current image data at different angles outside the vehicle body; the data transmission line is used to transmit the current image data collected by the plurality of cameras to the control system; The control system is used to perform splicing processing on the received current image data and transmit the spliced ​​data to the visual display; the visual display is installed in the vehicle body, and the visual display is used to display auxiliary images outside the vehicle to the driver.

9. The forklift according to claim 8, characterized in that: A door is also installed on the vehicle body, and a distance sensor and an angle sensor are provided on the door. The distance sensor is used to detect the distance information between the vehicle body and an obstacle, and the angle sensor is used to detect the moving angle information of the vehicle body. The data transmission line is also used to transmit the distance information and the moving angle information to the control system.

10. The forklift according to claim 9, characterized in that: The cameras are all explosion-proof cameras, and three of them are provided. The corresponding viewing angle of each explosion-proof camera is 120 degrees.

Images