Locomotive auxiliary trailer system, method, device, equipment and storage medium

Abstract

The present application relates to a locomotive-assisted trailer system, method, device, equipment and storage medium. The system includes: a road condition sensor, which obtains the operating road condition image information in front of the locomotive; a coupler sensor, which obtains the coupler image information of the locomotive and the trailer; a radar sensor, which obtains the road condition point cloud data; a central processing unit, which is respectively connected to the road condition sensor, the coupler sensor and the radar sensor; the central processing unit identifies the type of the trailer according to the trailer image, and identifies the coupler state and the coupler distance according to the coupler image information, and determines the distance between the locomotive and the trailer according to the road condition point cloud data, and sends the trailer type, coupler state, coupler distance and / or vehicle distance to the vehicle-mounted terminal or intelligent driving system. Based on the above system, the driver or the intelligent driving system can control the locomotive to approach the trailer until the coupler is connected when the locomotive-assisted trailer system determines that the coupler state meets the trailer requirements, thereby improving the degree of intelligence.

Description

Technical Field

[0001] The present application relates to the field of train monitoring technology, and in particular to a locomotive auxiliary trailer system, method, device, equipment and storage medium. Background Art

[0002] With the development of intelligent driving technology, locomotive trailer operation is one of the key issues that need to be addressed. At present, trailer operation requires the driver to look out and judge the safety of walking, judge the distance through signal flags or hook video monitoring, and then operate the trailer based on the driver's experience. The entire operation process needs to rely on the driver's experience and the degree of intelligence is low.

[0003] That is, the current trailer operation has the problem of low intelligence level. Summary of the invention

[0004] Based on this, it is necessary to provide a locomotive-assisted trailer system, method, device, equipment and storage medium that can improve the intelligence level of trailer operation in response to the above-mentioned technical problems.

[0005] A locomotive auxiliary trailer system, comprising:

[0006] A road condition sensor is used to obtain the running road condition image information in front of the locomotive; the running road condition image information includes the image of the trailer in the same track;

[0007] Coupler sensor, used to obtain coupler image information of the locomotive and the trailer;

[0008] The radar sensor is installed on the front of the locomotive to obtain point cloud data of road conditions;

[0009] The central processing unit is connected to the road condition sensor, the coupler sensor and the radar sensor respectively; the central processing unit identifies the type of the trailer according to the trailer image, and identifies the coupler status and the coupler distance according to the coupler image information, and determines the distance between the locomotive and the trailer according to the road condition point cloud data, and sends the trailer type, coupler status, coupler distance and / or vehicle distance to the on-board terminal or intelligent driving system, and assists the locomotive in trailering.

[0010] In one embodiment, the radar sensor includes a millimeter wave radar and a laser radar, both of which are arranged on the front of the locomotive, the millimeter wave radar is used to obtain the first road condition point cloud data, and the laser radar is used to obtain the second road condition point cloud data;

[0011] The central processor obtains a first recognition result according to the first road condition point cloud data, obtains a second recognition result according to the second road condition point cloud data, and determines the distance between the locomotive and the trailer based on the trailer type, the first recognition result and the second recognition result.

[0012] In one of the embodiments, the operating road condition image information further includes a derailer image;

[0013] The central processing unit inputs the derailer image into a preset derailer visual model, determines the derailer position and derailer status, and generates a first auxiliary instruction according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

[0014] In one embodiment, the coupler state includes a coupler connection state;

[0015] The central processing unit, during trailer operation, inputs the coupler image information into a preset coupler visual model to identify the coupler connection status, and generates a second auxiliary instruction according to the corresponding coupler connection status; the second auxiliary instruction is used to instruct the vehicle-mounted terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

[0016] A locomotive-assisted trailer method, comprising:

[0017] Obtaining the running road condition image information in front of the locomotive, the image information of the locomotive and the hook of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0018] Identify the type of the trailer according to the trailer image;

[0019] According to the coupler image information, the coupler status and coupler spacing are identified;

[0020] Determine the distance between the locomotive and the trailer based on the road condition point cloud data;

[0021] The trailer type, coupler status, coupler spacing and / or vehicle distance are sent to the vehicle terminal or intelligent driving system and auxiliary locomotive for trailering.

[0022] In one of the embodiments, the traffic point cloud data includes first traffic point cloud data and second traffic point cloud data;

[0023] The step of determining the distance between the locomotive and the trailer according to the road condition point cloud data includes the following steps:

[0024] A first recognition result is obtained based on the first road condition point cloud data, and a second recognition result is obtained based on the second road condition point cloud data, and the distance between the locomotive and the trailer is determined based on the type of the trailer, the first recognition result, and the second recognition result.

[0025] In one of the embodiments, the operating road condition image information further includes a derailer image;

[0026] After the step of obtaining the running road condition image information in front of the locomotive, the locomotive and the hook image information of the trailer, and the road condition point cloud data, the following steps are also included:

[0027] The derailer image is input into a preset derailer visual model to determine the derailer position and derailer status, and a first auxiliary instruction is generated according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

[0028] In one embodiment, the coupler state includes a coupler connection state;

[0029] The step of sending the trailer type, coupler status, coupler spacing and / or vehicle spacing to the vehicle-mounted terminal or intelligent driving system to the auxiliary locomotive for trailering also includes the following steps:

[0030] During trailer operation, the coupler image information is input into a preset coupler visual model to identify the coupler connection status, and a second auxiliary instruction is generated according to the corresponding coupler connection status; the second auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

[0031] A locomotive auxiliary trailer device, comprising:

[0032] An acquisition module is used to acquire the image information of the running road condition in front of the locomotive, the image information of the coupler of the locomotive and the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0033] A trailer type recognition module, used to recognize the trailer type according to the trailer image;

[0034] A coupler identification module is used to identify the coupler status and coupler spacing based on the coupler image information;

[0035] The vehicle distance recognition module is used to determine the distance between the locomotive and the trailer based on the road condition point cloud data;

[0036] The auxiliary module sends the trailer type, coupler status, coupler spacing and / or vehicle distance to the on-board terminal or intelligent driving system and assists the locomotive in trailering.

[0037] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0039] One of the above technical solutions has the following advantages and beneficial effects:

[0040] The locomotive auxiliary trailer system includes a road condition sensor, a coupler sensor, a radar sensor and a central processing unit. The central processing unit identifies the type of trailer based on the trailer image, identifies the coupler status and coupler spacing based on the coupler image information, and determines the distance between the locomotive and the trailer based on the road condition point cloud data, and sends the trailer type, coupler status, coupler spacing and / or vehicle spacing to the vehicle-mounted terminal or intelligent driving system to assist the locomotive in trailering. Thus, the driver or intelligent driving system can control the locomotive according to the received trailer type and vehicle spacing, stop at a certain distance from the trailer, and then control the locomotive to approach the trailer based on the acquired coupler status and coupler spacing, when the locomotive auxiliary trailer system determines that the coupler status meets the trailer requirements, until the locomotive auxiliary trailer system determines that the coupler is in a connected state, so as to realize the trailer operation, effectively reduce the intensity of manual driving and observation work, and improve the degree of intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a structural schematic diagram of a locomotive auxiliary trailer system in one embodiment;

[0043] Figure 2 is a schematic diagram of the structure of a radar sensor in one embodiment;

[0044] Figure 3 is a first schematic flow chart of a locomotive-assisted trailer method in one embodiment;

[0045] Figure 4 A schematic flow chart of the steps of determining the distance between a locomotive and a trailer in one embodiment;

[0046] Figure 5 is a second schematic flow chart of a locomotive-assisted trailer method in one embodiment;

[0047] Figure 6 1 is a structural block diagram of a locomotive auxiliary trailer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0050] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0051] It should be noted that when an element is considered to be "connected" to another element, it can be directly connected to the other element, or connected to the other element through an intermediate element. In addition, the "connection" in the following embodiments should be understood as "electrical connection", "communication connection", etc. if there is transmission of electrical signals or data between the connected objects.

[0052] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0053] In one embodiment, Figure 1 As shown, a locomotive auxiliary trailer system is provided, comprising:

[0054] A road condition sensor is used to obtain the running road condition image information in front of the locomotive; the running road condition image information includes the image of the trailer in the same track;

[0055] Coupler sensor, used to obtain coupler image information of the locomotive and the trailer;

[0056] The radar sensor is installed on the front of the locomotive to obtain point cloud data of road conditions;

[0057] The central processing unit is connected to the road condition sensor, the coupler sensor and the radar sensor respectively; the central processing unit identifies the type of the trailer according to the trailer image, and identifies the coupler status and the coupler distance according to the coupler image information, and determines the distance between the locomotive and the trailer according to the road condition point cloud data, and sends the trailer type, coupler status, coupler distance and / or vehicle distance to the on-board terminal or intelligent driving system, and assists the locomotive in trailering.

[0058] The road condition sensor is used to obtain the road condition image information in front of the locomotive. The running road condition image information is the real-time image information of the track and trackside in front of the locomotive, including the image of the trailer in the same track. The road condition sensor can be one or more road condition cameras. Generally, the road condition camera is installed in the middle of the driver's station, close to the front windshield of the train.

[0059] The coupler sensor is used to obtain the image information of the locomotive coupler and the hook of the trailer in real time. The coupler sensor may include one or more coupler cameras. Generally, the coupler sensor is installed above the locomotive coupler to ensure that the coupler camera field of view can cover the locomotive coupler and the couplers of other vehicles within a certain range on the same track.

[0060] The radar sensor is installed on the front of the locomotive and transmits the road condition point cloud data acquired in real time to the central processor so that the central processor can detect the distance between the vehicle and the trailer. The radar sensor may include millimeter wave radar, laser radar, etc. to cover the blind spot. It should be understood that a variety of radar sensors can be configured according to the system robustness requirements, and the configurable radar detection distance should be no less than 100 meters.

[0061] The central processing unit is a device or component with data processing and control functions, which is connected to the road condition sensor, the coupling sensor and the radar sensor respectively. The central processing unit is also used to connect to the vehicle terminal so that the processing results can be prompted to the driver through the vehicle terminal to assist the driver in trailer operation; or the central processing unit is also used to connect to the intelligent driving system to directly output auxiliary result information to assist the intelligent driving system in operation. The vehicle terminal is an audio-visual device with a reminder function. The central processing unit can prompt the driver to perform the next operation through the vehicle terminal in the form of voice, text, and pictures.

[0062] Specifically, the central processor can identify the type of trailer in the same track as the current locomotive based on the trailer image, and determine the distance between the current locomotive and the trailer based on the road condition point cloud data. The type of trailer and the distance between the locomotive and the trailer identified by the central processor are sent to the vehicle terminal or intelligent driving system to guide the locomotive to stop within a certain distance of the trailer. At the same time, the trailer coupler is detected by the coupler camera. Once the trailer coupler enters the field of view of the coupler sensor, the locomotive auxiliary trailer system uses the state and distance measurement identified by the coupler sensor as the main detection basis. The central processor detects the coupler state and the coupler interval between the two vehicles based on the coupler image information of the locomotive and the trailer, and transmits the recognition result to the vehicle terminal or intelligent driving system. The coupler state may include the coupler force state and the coupler connection state. The coupler connection state includes the coupler opening and closing state and whether the coupler is in a connected state (i.e., the coupler of the vehicle is connected to the coupler of another vehicle). When the opening and closing status of the couplers of the two vehicles meets the requirements of trailering, the information that the couplers meet the conditions for coupling is transmitted to the external device, guiding the driver or the intelligent driving system to control the locomotive to approach the trailer for trailering. Furthermore, when the locomotive approaches the trailer, the locomotive auxiliary trailer system monitors the distance between the two couplers and the coupler status in real time through the coupler sensor, and outputs the coupler status information and distance information during coupling; when the coupler is identified as being coupled, the central processor sends the coupling information to the outside to complete the auxiliary trailering.

[0063] The above locomotive auxiliary trailer system includes a road condition sensor, a coupler sensor, a radar sensor and a central processing unit. Among them, the central processing unit is connected to the road condition sensor, the coupler sensor and the radar sensor respectively, and is also used to connect the vehicle terminal or the intelligent driving system. The central processing unit sends the identified trailer type, coupler status, coupler spacing and / or the distance between the two vehicles to the outside in real time, so as to guide the locomotive to stop within a certain range from the trailer. At the same time, the system detects the coupler of the trailer and the vehicle through the coupler sensor, identifies the coupler status and coupler spacing of the two vehicles, and guides the locomotive to approach the trailer through the vehicle terminal or the intelligent driving system when the coupler opening and closing state meets the trailer requirements. During the approaching process, the coupler spacing and coupler status are monitored in real time until the coupler is detected to be hooked up, and the connection information is sent to the outside to complete the auxiliary trailer, without the need to manually confirm the coupler status, coupler connection status, trailer type and vehicle spacing, etc., and intelligently assist the trailer operation. This system can effectively reduce the difficulty of trailer operation, especially for workers who lack trailer operation experience.

[0064] In one embodiment, after the locomotive auxiliary trailer system is installed, calibration is required due to the differences in the installation positions of the road condition sensor and the coupler sensor to unify the coordinate systems of the road condition sensor and the radar sensor for subsequent data calculation and target fusion; for the coupler sensor, calibration is performed based on the internal and external parameters of the camera, the type and size of the coupler, and the position in the camera to provide basic parameters for coupler status detection and distance judgment between couplers.

[0065] In one embodiment, Figure 2 As shown, the radar sensor includes a millimeter wave radar and a laser radar, both of which are arranged on the front of the locomotive, the millimeter wave radar is used to obtain the first road condition point cloud data, and the laser radar is used to obtain the second road condition point cloud data;

[0066] The central processor obtains a first recognition result according to the first road condition point cloud data, obtains a second recognition result according to the second road condition point cloud data, and determines the distance between the locomotive and the trailer based on the trailer type, the first recognition result and the second recognition result.

[0067] Among them, radar sensors include millimeter-wave radar and laser radar, both of which are installed on the front of the locomotive. The millimeter-wave radar emits a beam of electromagnetic waves, and the central processing unit calculates the distance between the current locomotive and the target obstacle by the difference between the emitted electromagnetic waves and the reflected electromagnetic waves. When the laser radar cooperates with the central processing unit, the central processing unit can calculate the distance between the locomotive and the trailer based on the length of time between the emitted laser and the reflected laser, the propagation speed of light, and the current vehicle speed.

[0068] Specifically, the millimeter wave radar is used to obtain the first road condition point cloud data, and the laser radar is used to obtain the second road condition point cloud data.

[0069] The CPU extracts the vehicle point cloud and track features through the first point cloud data according to the first road condition point cloud data, segments and clusters the point cloud according to the track and point cloud features, detects the obstacle target ahead and measures the distance, and obtains the first recognition result; at the same time, according to the second road condition point cloud data, the CPU extracts the vehicle point cloud and track features through the second point cloud data, segments and clusters the point cloud according to the track and point cloud features, detects the obstacle target ahead and measures the distance, and obtains the second recognition result. Furthermore, the CPU determines the type of trailer and the distance between the locomotive and the trailer through time and space matching based on the identified trailer type, the first recognition result and the second recognition result, and transmits the distance result between the locomotive and the trailer to the driver through the on-board terminal for reminder or to the intelligent driving system for auxiliary operation.

[0070] In the embodiment of the present application, the radar sensor includes a millimeter-wave radar and a laser radar, which determines the distance between the current locomotive and the trailer through the first road condition point cloud data and the second point cloud data. This can effectively avoid the situation where a single radar is affected by factors such as weather and affects the distance recognition result, thereby improving the recognition accuracy.

[0071] In one embodiment, the operating road condition image information further includes a derailer image;

[0072] The central processing unit inputs the derailer image into a preset derailer visual model, determines the derailer position and derailer status, and generates a first auxiliary instruction according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

[0073] Since the derailer is a small target, the existing technology does not have the function of controlling the locomotive to identify the state of the derailer during the process and then start and stop the locomotive. The requirement for the locomotive coupling operation is that the locomotive stops a certain distance in front of the derailer. The function of the derailer is to mark the front storage line as being in operation when it is up (red) and the locomotive cannot enter to avoid safety accidents; when the derailer is marked as down (green), it indicates that the front storage line is not in operation and the locomotive can enter for coupling operation.

[0074] Specifically, the running road condition image information also includes a derailer image, and the derailer image includes derailer status information. The central processing unit inputs the derailer image into a preset derailer visual model to determine the derailer position and derailer status in the derailer image. Then, the first auxiliary instruction is generated according to the derailer status to control the on-board terminal to prompt or instruct the intelligent driving system to control the locomotive to continue driving. Based on the received first auxiliary instruction, the driver or the intelligent driving system controls the locomotive to continue driving to ensure safety after confirming that the derailer is in the lower derailment state. Then, the road condition sensor in the locomotive auxiliary trailer system detects the trailer in the current driving track, detects the distance between the locomotive and the trailer by the radar sensor, and detects the coupler state and the coupler spacing by the coupler sensor to assist the driver or the intelligent driving system in trailer operation.

[0075] In one embodiment, the preset derailer visual model is obtained by the following steps:

[0076] Obtain multiple derailer images, annotate and extract the derailers in the derailer images, and then construct a training set and a test set;

[0077] Input a training set to train the derailer deep learning model, and obtain a trained derailer deep learning model;

[0078] The trained derailer deep learning model is tested through the test set to obtain the preset derailer visual model.

[0079] Specifically, after the derailer deep learning model is trained with the training set, the trained model is obtained, and the test set is used for testing. According to the test results, the sample training is supplemented in a targeted manner. This process is recursively executed until the test results meet the application indicators, and the preset derailer visual model is obtained to accurately identify the derailer as being in an upper or lower state. It should be noted that the training set and the test set need to cover derailer images in different seasons, meteorological conditions and time periods. And the training set and the test set are collected independently and do not overlap.

[0080] In one embodiment, the coupler state includes a coupler connection state;

[0081] The central processing unit, during trailer operation, inputs the coupler image information into a preset coupler visual model to identify the coupler connection status, and generates a second auxiliary instruction according to the corresponding coupler connection status; the second auxiliary instruction is used to instruct the vehicle-mounted terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

[0082] Specifically, after the central processor sends the trailer type and vehicle distance information to the vehicle-mounted terminal or intelligent driving system, when performing trailer operations, the coupler image information transmitted by the coupler sensor is input into the preset coupler visual model to identify the coupler connection state, and a second auxiliary instruction is generated according to the corresponding coupler connection state. The coupler connection state includes the coupler opening and closing state and whether the coupler is in a connected state. When it is recognized that the coupler opening and closing state meets the trailer requirements, a second auxiliary instruction is generated to instruct the vehicle-mounted terminal to prompt the driver or instruct the intelligent driving system to control the locomotive to approach the trailer. When it is recognized that the coupler is connected based on the coupler image information, a corresponding second auxiliary instruction is generated to remind the driver or inform the intelligent driving system to complete the auxiliary trailer, effectively improving the trailer efficiency and the intelligence level of the trailer.

[0083] In one embodiment, the preset coupler visual model is obtained by the following steps:

[0084] Obtain multiple coupler image information, annotate and extract the couplers in the coupler image information, and construct a training set and a test set;

[0085] Input the training set to train the coupler deep learning model, and obtain the trained coupler deep learning model;

[0086] The trained coupler deep learning model is tested through the test set to obtain the preset coupler vision model.

[0087] Specifically, the coupler image information is collected through the coupler sensor, the coupler status in the coupler image information is annotated and extracted, the coupler deep learning model is trained through the extracted training set samples, the trained model is tested using the test set, and the sample training is supplemented in a targeted manner according to the test results. This process is recursively executed until the test results meet the application indicators and the preset coupler vision model is obtained. The training set and the test set need to cover different time periods, and the training set and the test set are collected independently and do not overlap; the central processor completes the coupler position recognition, coupler opening and closing state recognition, or whether the coupler is in a connected state through the coupler machine vision model.

[0088] In one embodiment, Figure 3 As shown, a locomotive-assisted trailer method is provided, comprising:

[0089] Step S120, obtaining the running road condition image information in front of the locomotive, the image information of the coupler of the locomotive and the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0090] Step S140, identifying the type of the trailer according to the trailer image;

[0091] Step S160, identifying the coupler state and the coupler spacing according to the coupler image information;

[0092] Step S180, determining the distance between the locomotive and the trailer according to the road condition point cloud data;

[0093] Step S200: Send the trailer type, coupler status, coupler spacing and / or vehicle spacing to the vehicle-mounted terminal or intelligent driving system to assist the locomotive in trailering.

[0094] The running road condition image information is the real-time image information of the track and trackside in front of the locomotive, including the image of the trailer in the same track. The running road condition image information can be obtained by the road condition sensor. The image information of the locomotive and the trailer coupler includes the image of the locomotive and the trailer coupler, which can be obtained by the coupler sensor. The road condition point cloud data is the point cloud data in front of the locomotive collected by the radar sensor.

[0095] Specifically, the type of trailer in the same track as the current locomotive is identified based on the trailer image, and the distance between the current locomotive and the trailer is determined based on the road condition point cloud data. The identified trailer type and the distance between the locomotive and the trailer are sent to the vehicle terminal or intelligent driving system to guide the locomotive to stop within a certain distance of the trailer. At the same time, according to the image information of the hooks of the locomotive and the trailer, the hook state and the hook interval of the two vehicles are detected, and the identification results are transmitted to the vehicle terminal or intelligent driving system. The hook state may include the stress state of the hook and the hook connection state. The hook connection state includes the opening and closing state of the hook and whether the hook is connected (that is, the hook of the vehicle is connected to the hook of another vehicle). When the opening and closing state of the hooks of the two vehicles meets the trailer requirements, the information that the hook has the connection conditions is transmitted to the external device to guide the driver or the intelligent driving system to control the locomotive to approach the trailer for trailering. Furthermore, when the locomotive approaches the trailer, the coupler image information monitored in real time by the coupler sensor is used to identify the distance between the two couplers and the coupler status, and output the coupler status information and distance information. When the coupler is identified as being connected, the connection information is sent to the on-board terminal or intelligent driving system to complete the auxiliary trailer, effectively improving the degree of intelligence in the trailer operation process.

[0096] In the embodiment of the present application, a locomotive-assisted trailer method is provided, which can effectively improve the trailer operation efficiency and help improve the degree of intelligent operation. It should be noted that the locomotive-assisted trailer method can be executed by the central processing unit in the above embodiment, or by other devices, which is not specifically limited here.

[0097] In one embodiment, the traffic point cloud data includes first traffic point cloud data and second traffic point cloud data;

[0098] like Figure 4 As shown, the step of determining the distance between the locomotive and the trailer according to the road condition point cloud data includes the steps of:

[0099] Step S182: Obtain a first recognition result based on the first road condition point cloud data, obtain a second recognition result based on the second road condition point cloud data, and determine the distance between the locomotive and the trailer based on the trailer type, the first recognition result, and the second recognition result.

[0100] Specifically, the first point cloud data is acquired in real time by the millimeter wave radar, and the second point cloud data is acquired in real time by the laser radar. According to the first road condition point cloud data, the vehicle point cloud and track features are extracted through the first point cloud data, and the first point cloud data is segmented and clustered for distance measurement based on the track and point cloud features to obtain the first recognition result; at the same time, according to the second road condition point cloud data, the vehicle point cloud and track features are extracted through the second point cloud data, and the second point cloud data is segmented and clustered for the track and point cloud features, and the obstacle target in front of the locomotive is detected and measured to obtain the second recognition result. Then, according to the type of trailer obtained by recognition, the first recognition result and the second recognition result are matched in time and space, and the type of trailer and the distance between the locomotive and the trailer are determined by fusion, and the distance between the locomotive and the trailer is transmitted to the on-board terminal through the on-board terminal for reminder or transmitted to the intelligent driving system for auxiliary operation, and the recognition accuracy is improved by fusing the first point cloud data and the second point cloud data.

[0101] In one embodiment, the operating road condition image information further includes a derailer image;

[0102] like Figure 5 As shown, after the step of obtaining the running road condition image information in front of the locomotive, the image information of the locomotive and the hook of the trailer, and the road condition point cloud data, the following steps are also included:

[0103] Step S130: input the derailer image into a preset derailer visual model, determine the derailer position and derailer state, and generate a first auxiliary instruction according to the derailer state; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

[0104] Specifically, the running road condition image information also includes a derailer image, and the derailer image includes derailer status information. The derailer image is input into a preset derailer visual model to determine the derailer position and derailer status in the derailer image. Then, a first auxiliary instruction is generated according to the derailer status to control the on-board terminal to prompt or instruct the intelligent driving system to control the locomotive to continue driving. Based on the received first auxiliary instruction, the driver or the intelligent driving system controls the locomotive to continue driving to ensure safety after confirming that the derailer is in the lower derailment state. Subsequently, the type of trailer in the current driving track is identified according to the running road condition image information detected by the road condition sensor, the distance between the locomotive and the trailer is detected by the road condition point cloud data, and the coupler state and the coupler spacing are detected by the coupler image information to assist the driver or the intelligent driving system in trailer operation.

[0105] In one embodiment, the preset derailer visual model is obtained by the following steps:

[0106] Obtain multiple derailer images, annotate and extract the derailers in the derailer images, and then construct a training set and a test set;

[0107] Input a training set to train the derailer deep learning model, and obtain a trained derailer deep learning model;

[0108] The trained derailer deep learning model is tested through the test set to obtain the preset derailer visual model.

[0109] Specifically, after the derailer deep learning model is trained with the training set, the trained model is obtained, and the test set is used for testing. According to the test results, the sample training is supplemented in a targeted manner. This process is recursively executed until the test results meet the application indicators, and the preset derailer visual model is obtained to accurately identify the derailer as being in an upper or lower state. It should be noted that the training set and the test set need to cover derailer images in different seasons, meteorological conditions and time periods. And the training set and the test set are collected independently and do not overlap.

[0110] In one embodiment, the coupler state includes a coupler connection state;

[0111] The step of sending the trailer type, coupler status, coupler spacing and / or vehicle spacing to the vehicle-mounted terminal or intelligent driving system to the auxiliary locomotive for trailering also includes the following steps:

[0112] During trailer operation, the coupler image information is input into a preset coupler visual model to identify the coupler connection status, and a second auxiliary instruction is generated according to the corresponding coupler connection status; the second auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

[0113] Specifically, the coupler image information transmitted by the coupler sensor is input into the preset coupler visual model to identify the coupler connection state, and a second auxiliary instruction is generated according to the corresponding coupler connection state. The coupler connection state includes the coupler opening and closing state and whether the coupler is in a connected state. When it is recognized that the coupler opening and closing state meets the trailer requirements, a second auxiliary instruction is generated to instruct the on-board terminal to prompt the driver or instruct the intelligent driving system to control the locomotive to approach the trailer. When it is recognized that the coupler is connected based on the coupler image information, a corresponding second auxiliary instruction is generated to remind the driver or inform the intelligent driving system to complete the auxiliary trailer, effectively improving the trailer efficiency and the intelligence level of the trailer.

[0114] In one embodiment, the preset coupler visual model is obtained by the following steps:

[0115] Obtain multiple coupler image information, annotate and extract the couplers in the coupler image information, and construct a training set and a test set;

[0116] Input the training set to train the coupler deep learning model, and obtain the trained coupler deep learning model;

[0117] The trained coupler deep learning model is tested through the test set to obtain the preset coupler vision model.

[0118] Specifically, the coupler image information is collected through the coupler sensor, the coupler status in the coupler image information is annotated and extracted, the coupler deep learning model is trained through the extracted training set samples, the trained model is tested using the test set, and the sample training is supplemented in a targeted manner according to the test results. This process is recursively executed until the test results meet the application indicators and the preset coupler vision model is obtained. The training set and the test set need to cover different time periods, and the training set and the test set are collected independently and do not overlap; the central processor completes the coupler position recognition, coupler opening and closing state recognition, or whether the coupler is in a connected state through the coupler machine vision model.

[0119] It should be understood that although Figure 3-Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 3-Figure 5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0120] In one embodiment, Figure 6 As shown, a locomotive auxiliary trailer device is provided, comprising: an acquisition module, a trailer type identification module, a coupler identification module, a vehicle distance identification module and an auxiliary module, wherein:

[0121] An acquisition module is used to acquire the running road condition image information in front of the locomotive, the image information of the locomotive and the coupler of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0122] A trailer type identification module, used to identify the type of the trailer according to the trailer image;

[0123] A coupler identification module, used to identify the coupler state and the coupler spacing according to the coupler image information;

[0124] A vehicle distance recognition module, used to determine the vehicle distance between the locomotive and the trailer according to the road condition point cloud data;

[0125] The auxiliary module sends the trailer type, the coupler status, the coupler spacing and / or the vehicle spacing to the vehicle-mounted terminal or the intelligent driving system to assist the locomotive in trailering.

[0126] For the specific definition of the locomotive auxiliary trailer device, please refer to the definition of the locomotive auxiliary trailer method above, which will not be repeated here. Each module in the above-mentioned locomotive auxiliary trailer device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0127] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0128] Obtaining the running road condition image information in front of the locomotive, the image information of the locomotive and the hook of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0129] Identify the type of the trailer according to the trailer image;

[0130] According to the coupler image information, the coupler status and coupler spacing are identified;

[0131] Determine the distance between the locomotive and the trailer based on the road condition point cloud data;

[0132] The trailer type, coupler status, coupler spacing and / or vehicle distance are sent to the vehicle terminal or intelligent driving system and auxiliary locomotive for trailering.

[0133] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0134] Obtaining the running road condition image information in front of the locomotive, the image information of the locomotive and the hook of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer in the same track;

[0135] Identify the type of the trailer according to the trailer image;

[0136] According to the coupler image information, the coupler status and coupler spacing are identified;

[0137] Determine the distance between the locomotive and the trailer based on the road condition point cloud data;

[0138] The trailer type, coupler status, coupler spacing and / or vehicle distance are sent to the vehicle terminal or intelligent driving system and auxiliary locomotive for trailering.

[0139] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0140] In the description of this specification, the description with reference to the terms "some embodiments", "other embodiments", "ideal embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0141] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A locomotive auxiliary trailer system, It is characterized in that include: A road condition sensor is used to obtain image information of the running road condition in front of the locomotive; The running road condition image information includes an image of a trailer and an image of a derailer in the same track; Coupler sensor, used to obtain coupler image information of the locomotive and the trailer; A radar sensor is arranged at the front of the locomotive, and is used to obtain road condition point cloud data; the radar sensor includes a millimeter wave radar and a laser radar, and the millimeter wave radar and the laser radar are both arranged at the front of the locomotive, the millimeter wave radar is used to obtain first road condition point cloud data, and the laser radar is used to obtain second road condition point cloud data; the road condition sensor and the radar sensor adopt a unified coordinate system; a central processing unit connected to the road condition sensor, the coupler sensor and the radar sensor respectively; the central processing unit identifies the type of the trailer according to the trailer image, identifies the coupler state and the coupler distance according to the coupler image information, determines the distance between the locomotive and the trailer according to the road condition point cloud data, sends the trailer type, the coupler state, the coupler distance and / or the distance to the vehicle terminal or the intelligent driving system, and assists the locomotive in trailering; the central processing unit extracts the vehicle point cloud and track features according to the first road condition point cloud data, segments and clusters the point cloud according to the track and point cloud features, detects the obstacle target in front of the running and measures the distance, and obtains a first recognition result; extracts the vehicle point cloud and track features according to the second road condition point cloud data, segments and clusters the point cloud according to the track and point cloud features, detects the obstacle target in front and measures the distance, and obtains a second recognition result, and determines the distance between the locomotive and the trailer based on the type of the trailer, the first recognition result and the second recognition result; The central processor also inputs the derailer image into a preset derailer visual model, determines the derailer position and derailer status, and generates a first auxiliary instruction according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive; the coupler status includes the coupler connection status; the central processor, when operating the trailer, inputs the coupler image information into a preset coupler visual model, identifies the coupler connection status, and generates a second auxiliary instruction according to the corresponding coupler connection status; the second auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

2. A locomotive-assisted trailer method, It is characterized in that Applied to the locomotive auxiliary trailer system of claim 1, the method comprises: Obtaining the running road condition image information in front of the locomotive, the image information of the locomotive and the coupler of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer and the image of the derailer in the same track; the road condition point cloud data includes the first road condition point cloud data and the second road condition point cloud data; the first road condition point cloud data is obtained in real time by the millimeter wave radar, and the second road condition point cloud data is obtained in real time by the laser radar; identifying the type of the trailer according to the trailer image; According to the coupler image information, identifying the coupler state and the coupler spacing; Determining the distance between the locomotive and the trailer according to the road condition point cloud data, including: extracting vehicle point cloud and track features according to the first road condition point cloud data, segmenting and clustering the first road condition point cloud data for distance measurement based on the track and point cloud features, and obtaining a first recognition result; and extracting vehicle point cloud and track features according to the second road condition point cloud data, segmenting and clustering the second road condition point cloud data for distance measurement based on the track and point cloud features, and obtaining a second recognition result, and determining the distance between the locomotive and the trailer based on the type of the trailer, the first recognition result, and the second recognition result; Sending the trailer type, the coupler status, the coupler spacing and / or the vehicle spacing to an on-board terminal or an intelligent driving system to assist the locomotive in trailering; The derailer image is input into a preset derailer visual model to determine the derailer position and derailer status, and a first auxiliary instruction is generated according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

3. The locomotive-assisted trailer method according to claim 2, It is characterized in that The coupler state includes a coupler connection state; The step of sending the trailer type, the coupler state, the coupler spacing and / or the vehicle spacing to the vehicle-mounted terminal or the intelligent driving system to assist the locomotive in trailering also includes the following steps: During trailer operation, the coupler image information is input into a preset coupler visual model to identify the coupler connection state, and a second auxiliary instruction is generated according to the corresponding coupler connection state; the second auxiliary instruction is used to instruct the vehicle-mounted terminal to give a prompt or instruct the intelligent driving system to complete the trailer operation.

4. A locomotive auxiliary trailer device, It is characterized in that The locomotive auxiliary trailer system according to claim 1, wherein the device comprises: An acquisition module is used to acquire the running road condition image information in front of the locomotive, the image information of the locomotive and the coupler of the trailer, and the road condition point cloud data; the running road condition image information includes the image of the trailer and the derailer in the same track; A trailer type identification module, used to identify the type of the trailer according to the trailer image; A coupler identification module, used to identify the coupler state and the coupler spacing according to the coupler image information; A vehicle distance recognition module is used to determine the vehicle distance between the locomotive and the trailer according to the road condition point cloud data; the road condition point cloud data includes first road condition point cloud data and second road condition point cloud data; the first road condition point cloud data is acquired in real time by a millimeter wave radar, and the second road condition point cloud data is acquired in real time by a laser radar; the vehicle distance recognition module is also used to extract vehicle point cloud and track features according to the first road condition point cloud data, segment and cluster the first road condition point cloud data for distance measurement according to the track and point cloud features, and obtain a first recognition result; and extract vehicle point cloud and track features according to the second road condition point cloud data, segment and cluster the second road condition point cloud data for distance measurement according to the track and point cloud features, and obtain a second recognition result, and determine the vehicle distance between the locomotive and the trailer based on the type of the trailer, the first recognition result, and the second recognition result; An auxiliary module, which sends the trailer type, the coupler status, the coupler spacing and / or the vehicle spacing to an on-board terminal or an intelligent driving system to assist the locomotive in trailering; The auxiliary module is also used to input the derailer image into a preset derailer visual model, determine the derailer position and derailer status, and generate a first auxiliary instruction according to the derailer status; the first auxiliary instruction is used to instruct the on-board terminal to give a prompt or instruct the intelligent driving system to control the travel of the locomotive.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor executes the computer program, the steps of the method according to claim 2 or 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 2 or 3 are implemented.

Images