A panoramic camera image fusion method and system for ship berthing

Abstract

The present disclosure relates to a method and system for fusion of panoramic camera images for ship berthing, and relates to image processing technology; obtaining a camera image captured by a target camera through the direction of the ship compared to the berthing position; determining a target ship image and a target berthing environment image through berthing position information and a pre-configured image set; and then, performing image fusion using the camera image, the target ship image, and the berthing environment image to obtain a target panoramic image. In this technical solution, the camera image and the pre-configured image set are combined, and the berthing position information is used to perform image fusion, which reduces the requirements for the image acquisition accuracy of the camera and the camera setting method, and does not require complex image fusion technology, so that simple, effective and accurate panoramic image acquisition for berthing can be achieved.

Description

Technical Field

[0001] The present disclosure relates to image processing technology, and in particular to a method and system for fusing panoramic camera images for ship berthing. Background Art

[0002] With the development of image processing technology, image processing technology has been applied to various application scenarios. In the ship berthing scenario, panoramic images can be synthesized through image processing technology so that the panoramic images can assist the ship in berthing operations.

[0003] In the related art, panoramic images of ships at berthing are usually obtained by fusing images acquired in real time by multiple cameras. This method places high demands on the image acquisition accuracy of the cameras and the camera settings. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method and system for fusion of panoramic camera images for ship berthing, which can realize simple, effective and accurate acquisition of panoramic images for berthing without complicated image fusion technology.

[0005] In order to achieve the above-mentioned objectives, in a first aspect, the present disclosure provides a panoramic camera image fusion method for ship berthing, comprising: obtaining berthing position information of the ship, the berthing position information comprising the ship coordinates of the ship in a coordinate system with the berthing position as the coordinate origin and the direction of the ship relative to the berthing position; obtaining a camera image captured by a target camera of the ship according to the direction of the ship relative to the berthing position; determining a target ship image and a target berthing environment image according to the berthing position information and a pre-configured image set, the pre-configured image set comprising: a plurality of ship images and berthing environment images respectively corresponding to the berthing position information; performing image fusion according to the camera image, the target ship image and the target berthing environment image to obtain a target panoramic image.

[0006] Optionally, the ship includes cameras arranged in multiple directions respectively, and the direction of the ship compared to the berthing position is the direction of the bow of the ship compared to the berthing position, and obtaining the camera image captured by the target camera of the ship according to the orientation of the ship compared to the berthing position includes: determining the target direction having the smallest directional angle with the direction of the bow of the ship compared to the berthing position from the multiple directions; and obtaining the camera image captured by the target camera arranged in the target direction of the ship.

[0007] Optionally, the preconfigured image set includes a ship image set and a berthing environment image set, the ship image set includes ship images corresponding to multiple preset ship coordinates, the berthing environment image set includes berthing environment images corresponding to multiple preset ship coordinates, and determining the target ship image and the target berthing environment image according to the berthing position information and the preconfigured image set includes: determining the target ship image according to the ship images corresponding to the multiple preset ship coordinates, the ship coordinates, and the direction of the ship compared to the berthing position; determining the target berthing environment image according to the berthing environment images corresponding to the multiple preset ship coordinates and the ship coordinates.

[0008] Optionally, determining the target ship image based on the ship images respectively corresponding to the multiple preset ship coordinates, the ship coordinates and the direction of the ship compared to the berthing position includes: determining a first ship image based on the ship images respectively corresponding to the multiple preset ship coordinates and the ship coordinates; determining a second ship image based on the first ship image, the direction of the ship compared to the berthing position and a pre-trained image transformation model; determining the target ship image based on the first ship image and the second ship image.

[0009] Optionally, the panoramic camera image fusion method for ship berthing also includes: obtaining a first training data set, the first training data set including multiple first training samples, each first training sample including: a sample ship image, a sample ship direction corresponding to the sample ship image, and a transformed image corresponding to the sample ship image, wherein the similarity between the sample ship images in different first training samples is less than a preset similarity, and / or the sample ship directions corresponding to the sample ship images in different first training samples are different; training the image transformation model to be trained according to the first training data set to obtain the pre-trained image transformation model, and the pre-trained image transformation model is used to output the second ship image according to the input first ship image and the direction of the ship compared to the berthing position.

[0010] Optionally, the ship coordinates include a first ship coordinate and a second ship coordinate, the ship positions corresponding to the first ship coordinates and the second ship coordinates are relative ship positions, and determining the target berthing environment image according to the berthing environment images respectively corresponding to the multiple preset ship coordinates and the ship coordinates includes: determining the first berthing environment image corresponding to the first ship coordinates from the berthing environment images respectively corresponding to the multiple preset ship coordinates; determining the second berthing environment image corresponding to the second ship coordinates from the berthing environment images respectively corresponding to the multiple preset ship coordinates; and deduplicating the first berthing environment image and the second berthing environment image to obtain the target berthing environment image.

[0011] Optionally, performing image fusion according to the camera image, the target ship image and the target berthing environment image includes: respectively determining direction labels corresponding to the camera image, the target ship image and the target berthing environment image; wherein the direction label corresponding to the camera image is a first direction, the direction label corresponding to the target ship image is a second direction, and the direction labels corresponding to the target berthing environment image are a third direction and a fourth direction; inputting a pre-trained image fusion model according to the camera image, the target ship image, the target berthing environment image, the direction label corresponding to the camera image, the direction label corresponding to the target ship image and the direction label corresponding to the target berthing environment image, to obtain the target panoramic image output by the pre-trained image fusion model.

[0012] Optionally, the panoramic camera image fusion method for ship berthing also includes: obtaining a second training data set, the second training data set includes multiple second training samples, each second training sample includes: a sample camera image, a direction label corresponding to the sample camera image, a sample ship image, a direction label corresponding to the sample ship image, a sample berthing environment image, a direction label corresponding to the sample berthing environment image and a sample panoramic image, wherein different second training samples correspond to different sample panoramic images, and the sample camera image, the sample ship image and the sample berthing environment image are images obtained by splitting the sample panoramic image according to the corresponding direction labels; training the image fusion model to be trained according to the second training data set to obtain the pre-trained image fusion model.

[0013] Optionally, the method for fusing panoramic camera images for ship berthing also includes: acquiring historical ship berthing data, the historical ship berthing data including panoramic images for berthing and actual ship berthing paths corresponding to the panoramic images for berthing; comparing the target panoramic image with the panoramic image for berthing to obtain a comparison result; determining a recommended ship berthing path based on the comparison result and the historical ship berthing data; and displaying the target panoramic image and the recommended ship berthing path.

[0014] In a second aspect, the present disclosure provides a panoramic camera image fusion system for ship berthing, comprising: a plurality of cameras, respectively arranged in multiple directions of the ship, for collecting camera images; a ship control system, respectively connected to the plurality of cameras, for implementing the panoramic camera image fusion method for ship berthing as described in the first aspect.

[0015] Through the above technical solution, the berthing position information of the ship is obtained, which includes the coordinates of the ship in the coordinate system with the berthing position as the coordinate origin and the direction of the ship relative to the berthing position. The camera image captured by the target camera is obtained through the direction of the ship relative to the berthing position; the target ship image and the target berthing environment image are determined through the berthing position information and the pre-configured image set; then, the camera image, the target ship image and the berthing environment image are used for image fusion to obtain the target panoramic image. In this technical solution, the camera image and the pre-configured image set are combined with the berthing position information to perform image fusion, which reduces the requirements for the image acquisition accuracy of the camera and the camera setting method, and does not require complex image fusion technology, so that simple, effective and accurate panoramic image acquisition for berthing can be achieved.

[0016] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0018] Figure 1 The present invention is a block diagram of a panoramic camera image fusion system for ship berthing according to an exemplary embodiment.

[0019] Figure 2 The figure is a schematic diagram of a camera setting direction according to an exemplary embodiment.

[0020] Figure 3 The present invention is a flowchart of a method for fusing panoramic camera images for ship berthing according to an exemplary embodiment.

[0021] Figure 4 The figure is an example diagram of berthing position information according to an exemplary embodiment.

[0022] Figure 5 The figure is a diagram showing an example of a direction label according to an exemplary embodiment.

[0023] Figure 6 The present invention is a structural block diagram of a panoramic camera image fusion device for ship berthing according to an exemplary embodiment.

[0024] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0026] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0027] In the present disclosure, unless otherwise specified, directional words such as "up, down, left, right, front, back", etc. are used only to facilitate the description of the present disclosure and simplify the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore should not be understood as a limitation on the present disclosure.

[0028] With the development of image processing technology, image processing technology has been applied to various application scenarios. In the ship berthing scenario, panoramic images can be synthesized through image processing technology so that the panoramic images can assist the ship in berthing operations.

[0029] In the related art, panoramic images of ships at berthing are usually obtained by fusing images acquired in real time by multiple cameras. This method places high demands on the image acquisition accuracy of the cameras and the camera settings.

[0030] Based on this, the embodiment of the present disclosure provides a technical solution, in which the camera's captured image and a pre-configured image set are combined and image fusion is performed using the docking position information, which reduces the requirements for the camera's image acquisition accuracy and the camera's setting method. In addition, there is no need for complex image fusion technology, and simple, effective and accurate panoramic image acquisition for docking can be achieved.

[0031] Figure 1is a block diagram of a panoramic camera image fusion system for ship berthing according to an exemplary embodiment. Figure 1 The system comprises: a plurality of cameras and a ship control system.

[0032] In some embodiments, multiple cameras are respectively arranged in multiple directions of the ship to collect camera images, which may be images of the surrounding environment. Figure 1 The number of cameras shown in the figure does not constitute a limitation on the embodiments of the present disclosure.

[0033] For example, Figure 2 is a schematic diagram of a camera setting direction according to an exemplary embodiment. Figure 2 As shown, the ship can be divided into the bow direction, the stern direction and the hull direction, wherein the hull direction can include two sides, each of which corresponds to a respective direction. Thus, the camera can include multiple cameras arranged in the bow direction, the stern direction and the hull direction.

[0034] In some embodiments, the ship control system may be a central control system of the ship, or a navigation system, etc., which is not limited here. In other words, the panoramic camera image fusion system for berthing a ship is a part of the central control system of the ship.

[0035] Therefore, the images collected by multiple cameras can be used not only for panoramic image fusion, but also for other functions of the central control system.

[0036] Figure 3 FIG. 1 is a flow chart of a method for fusing panoramic camera images for berthing a ship according to an exemplary embodiment. Figure 3 As shown, the panoramic camera image fusion method for ship berthing includes the following steps:

[0037] Step S31, obtaining the berthing position information of the ship, the berthing position information including the ship coordinates in a coordinate system with the berthing position as the coordinate origin and the direction of the ship relative to the berthing position.

[0038] Step S32, obtaining a camera image captured by a target camera of the ship according to the direction of the ship relative to the berthing position.

[0039] Step S33, determining a target ship image and a target berthing environment image according to the berthing position information and a pre-configured image set, wherein the pre-configured image set includes: a plurality of ship images and berthing environment images corresponding to the berthing position information respectively.

[0040] Step S34, performing image fusion according to the camera image, the target ship image and the berthing environment image to obtain a target panoramic image.

[0041] In step S31, the berthing position information may be information obtained by positioning the ship when the ship is approaching the berthing position or heading for the berthing position.

[0042] In some embodiments, the ship may be positioned first, and then the ship coordinates in a coordinate system with the berthing position as the coordinate origin may be determined based on the ship positioning result. The ship positioning method, such as gyroscope positioning, wireless radar positioning, etc., may be specifically referred to mature technologies in the art and will not be described in detail here.

[0043] In some embodiments, the ship coordinates may be a coordinate set or a single coordinate. If it is a coordinate set, it may be expressed as, for example: {(x1, y1),…, (xn, yn)}, where different coordinates may represent the coordinates of different ship reference points, for example: (x1, y1) is the coordinate of the bow reference point, (xn, yn) is the coordinate of the stern reference point, and the specific setting method of the reference point may vary depending on the application scenario. If it is a single coordinate, it may be the coordinate of the bow position of the ship, for example (xm, ym), where m represents the bow position of the ship.

[0044] In some embodiments, based on the positioning result of the ship, the direction of the ship relative to the berthing position can also be determined. It can be understood that the positioning result usually includes a position and a direction, so the direction information can be determined based on the positioning result.

[0045] In some embodiments, the direction of the ship relative to the berthing position can be formed by taking the reference point of the ship as the starting point and the berthing position as the end point to form a direction vector, and the direction vector is used to represent the direction.

[0046] In some embodiments, other direction collectors may be used to obtain the direction of the ship relative to the berthing position. For details, reference may be made to mature technologies in the art and are not limited here.

[0047] Figure 4 is an example diagram showing a docking position information according to an exemplary embodiment. Figure 4 As shown, in a coordinate system with the berthing position as the coordinate origin, the ship coordinates (xm, ym) can be determined, and the vector I representing the direction of the ship relative to the berthing position can be determined. m-o , assuming the basis vector is O 0-x , then the vector I m-o With vector O 0-x The angle between them can also represent the corresponding direction.

[0048] In step S32, a camera image captured by a target camera of the ship is obtained according to the direction of the ship relative to the berthing position.

[0049] It is understandable that the ship is provided with multiple cameras, and the images collected by the multiple cameras can be applied in different scenes. Therefore, the image of the camera in the appropriate direction can be selected for application based on the actual direction.

[0050] As an optional embodiment, the direction of the ship compared to the berthing position is the direction of the bow of the ship compared to the berthing position. In this case, step S32 includes: determining the target direction with the smallest directional angle between the direction of the bow of the ship compared to the berthing position from multiple directions; obtaining a camera image captured by a target camera set in the target direction of the ship.

[0051] In some embodiments, direction vectors corresponding to multiple directions are generated, and then the angles between these multiple direction vectors and the direction vector of the bow of the ship compared to the berthing position are determined to obtain multiple direction angles, from which the minimum direction angle is determined, and the camera in the direction corresponding to the minimum direction angle is determined as the target camera.

[0052] It can be understood that the image acquisition direction of the target camera is consistent with its setting direction, so its image acquisition direction is closest to the direction of the ship compared to the berthing position. Therefore, the image captured by the target camera can be understood as an image captured in the direction of the berthing position.

[0053] In some embodiments, if the reference point is not the bow position but the stern position, the target camera is determined according to the same implementation as above.

[0054] In some embodiments, the selection of the reference point may be determined by the berthing method of the ship. For example, if the ship berths based on the bow, the reference point is the bow; if the ship berths based on the stern, the reference point is the stern; if the ship berths based on the hull, the reference point is the corresponding side of the hull. Therefore, the above implementation is only an example and can be flexibly adapted in actual application scenarios.

[0055] In step S33, the target ship image and the target berthing environment image are determined according to the berthing position information and the pre-configured image set.

[0056] In some embodiments, the preconfigured image set includes: ship images and berthing environment images corresponding to multiple berthing position information respectively. Specifically, the preconfigured image set includes a ship image set and a berthing environment image set. The ship image set includes ship images corresponding to multiple preset ship coordinates, and the berthing environment image set includes berthing environment images corresponding to multiple preset ship coordinates.

[0057] In some embodiments, various berthing conditions of the ship may be simulated in advance, and in the simulated berthing conditions, images of the ship and berthing environment may be collected at different ship positions (ship coordinates) to generate an image set.

[0058] In some embodiments, the collection direction of the ship image and the berthing environment image can be determined based on the berthing mode. For example, if the ship is berthing at the bow, the collection direction can be other directions except the bow direction. If the ship is berthing at the stern, the collection direction can be other directions except the stern direction. If the ship is berthing at the hull, the collection direction can be other directions except the berthing side direction of the hull.

[0059] Through this implementation, the camera image, the ship image and the berthing environment image can correspond to different directions respectively, thereby realizing the fusion of the panoramic image.

[0060] As an optional implementation, step S33 includes: determining the target ship image based on the ship images, ship coordinates and the direction of the ship relative to the berthing position corresponding to multiple preset ship coordinates; determining the target berthing environment image based on the berthing environment images and ship coordinates corresponding to multiple preset ship coordinates.

[0061] In some embodiments, a target ship image is determined based on ship images, ship coordinates, and the direction of the ship relative to the berthing position that respectively correspond to a plurality of preset ship coordinates, including: determining a first ship image based on ship images and ship coordinates that respectively correspond to a plurality of preset ship coordinates; determining a second ship image based on the first ship image, the direction of the ship relative to the berthing position, and a pre-trained image transformation model; and determining a target ship image based on the first ship image and the second ship image.

[0062] In some embodiments, the current ship coordinates are compared with a plurality of preset ship coordinates to determine the preset ship coordinates that match the current ship coordinates, and the ship image corresponding to the preset ship coordinates is determined as the first ship image.

[0063] In the aforementioned embodiment, it is introduced that the ship coordinates may be multiple coordinates based on multiple reference points, or may be one reference point coordinate. Correspondingly, the configuration method of multiple preset ship coordinates is similar, that is, if the ship coordinates are one reference point coordinates, then each preset ship coordinate is the corresponding reference point coordinate. In the case of one, the current ship coordinates match the preset ship coordinates, and the two coordinates are regarded as the same. In the case of multiple, if there are more than a preset number of reference points corresponding to the same ship coordinates, then the current ship coordinates are regarded as matching the preset ship coordinates. Among them, the preset number can be configured according to the number of reference points, for example: the preset number is half of the number of reference points.

[0064] Further, a second ship image is determined based on the first ship image, the direction of the ship relative to the berthing position, and a pre-trained image transformation model.

[0065] In this embodiment, the image transformation model can be understood as an extended model of the image, which can perform image transformation on the basis of the first ship image and the corresponding direction to obtain more images, thereby avoiding the problem of poor accuracy of the target ship image due to the singleness of the first ship image.

[0066] Therefore, as an optional implementation, the panoramic camera image fusion method for ship berthing also includes: obtaining a first training data set, the first training data set includes multiple first training samples, each first training sample includes: a sample ship image, a sample ship direction corresponding to the sample ship image, and a transformed image corresponding to the sample ship image, wherein the similarity between the sample ship images in different first training samples is less than a preset similarity, and / or the sample ship directions corresponding to the sample ship images in different first training samples are different; training the image transformation model to be trained according to the first training data set to obtain a pre-trained image transformation model, the pre-trained image transformation model is used to output a second ship image based on the input first ship image and the direction of the ship compared to the berthing position.

[0067] In this embodiment, the sample ship image may be an image obtained by collecting ship images in a reference direction, and the transformed image corresponding to the sample ship image may be a ship image collected after the ship is offset in the sample ship direction based on the ship position corresponding to the sample ship image.

[0068] In the disclosed embodiment, the ship image may be collected by using equipment such as drones and camera devices at berthing locations.

[0069] In some embodiments, the similarity between sample ship images in different first training samples is less than a preset similarity, which can be understood as the sample ship images can correspond to different ship states or ship berthing modes, etc., to ensure the richness of the samples. The sample ship images in different first training samples correspond to different sample ship directions, which can be understood as different offset directions, to ensure the richness of the samples.

[0070] Furthermore, since the difference between the sample ship image and the transformed image corresponding to the sample ship image is caused by the sample ship direction corresponding to the sample ship image, the model is trained by the first training data set so that the model can learn the image feature change characteristics caused by the direction offset. Therefore, after the image transformation model to be trained is trained according to the first training data set to obtain the pre-trained image transformation model, the pre-trained image transformation model can be used to output the second ship image according to the input first ship image and the direction of the ship compared to the berthing position.

[0071] In some embodiments, the type of the image transformation model can be a deep learning model, a large model, etc., a model with strong image processing capabilities.

[0072] In some embodiments, the number of the output second ship images may be one or more, which is not limited herein.

[0073] Furthermore, the first ship image and the second ship image can be directly determined as the target ship image; the first ship image and the second ship image can also be spliced, or feature points are screened to generate a new target ship image. For example, the repeated image feature parts in the first ship image and the second ship image are removed, and then the splicing operation is performed, so that the target ship image contains more image features.

[0074] Therefore, the target ship image can represent the ship image collected along the direction of the ship compared to the berthing position, which can replace the ship image collected by the camera.

[0075] In some embodiments, when the ship coordinates are a reference point, the berthing environment images corresponding to the preset ship coordinates include environment images in two relative directions. Then, when determining the target berthing environment image, the preset ship coordinates matching the ship coordinates are determined, and the berthing environment images in two relative directions corresponding to the preset ship coordinates are determined as the target berthing environment image.

[0076] In some embodiments, when the ship coordinates are multiple reference points, the berthing environment image corresponding to the preset ship coordinates includes only one, and the reference points corresponding to different preset ship coordinates may be the same or different.

[0077] Therefore, the configurations for the ship image set and the berthing environment image set are different and can be configured separately when setting the image set.

[0078] Furthermore, as an optional implementation, the ship coordinates include a first ship coordinate and a second ship coordinate, and the ship positions corresponding to the first ship coordinate and the second ship coordinate are relative ship positions. This implementation corresponds to the case where the aforementioned ship coordinates are multiple reference points, and multiple ship coordinates need to be collected accordingly.

[0079] Then, correspondingly, according to the berthing environment images and the ship coordinates respectively corresponding to the multiple preset ship coordinates, the target berthing environment image is determined, including: determining a first berthing environment image corresponding to the first ship coordinates from the berthing environment images respectively corresponding to the multiple preset ship coordinates; determining a second berthing environment image corresponding to the second ship coordinates from the berthing environment images respectively corresponding to the multiple preset ship coordinates; and deduplicating the first berthing environment image and the second berthing environment image to obtain the target berthing environment image.

[0080] In this embodiment, corresponding berthing environment images are determined based on the first ship coordinates and the second ship coordinates respectively to obtain a target berthing environment image.

[0081] It can be understood that as to whether the ship coordinates match, reference can be made to the introduction of the aforementioned embodiment.

[0082] Therefore, the target berthing environment image may correspond to two directions, and these two directions are different from the directions of the aforementioned camera image and ship image.

[0083] In step S34, image fusion is performed according to the camera image, the target ship image and the berthing environment image. As an optional implementation, step S34 includes: respectively determining the direction labels corresponding to the camera image, the target ship image and the target berthing environment image; wherein the direction label corresponding to the camera image is the first direction, the direction label corresponding to the target ship image is the second direction, and the direction labels corresponding to the target berthing environment image are the third direction and the fourth direction; inputting the pre-trained image fusion model according to the camera image, the target ship image, the berthing environment image, the direction label corresponding to the camera image, the direction label corresponding to the target ship image and the direction label corresponding to the target berthing environment image, obtaining the target panoramic image output by the pre-trained image fusion model.

[0084] In some embodiments, the direction label corresponding to the camera image is the first direction, which may be an image captured in the berthing direction, and may correspond to the environment in the berthing direction. The direction label corresponding to the target ship image is the second direction, which may be an image captured in the direction of the ship reference point (a reference point determined based on the berthing method), and may correspond to the ship in the direction of the ship reference point. The direction labels corresponding to the target berthing environment image are the third direction and the fourth direction, which may be images captured in directions other than the berthing direction and the ship reference point direction, and may correspond to the environment in non-berthing directions.

[0085] Figure 5 is an example diagram of a direction label according to an exemplary embodiment. Figure 5 As shown, assuming that the direction of the ship reference point is the bow, the first direction is S1 in the figure, the second direction is S2 in the figure, and the third direction and the fourth direction are S3 and S4 in the figure.

[0086] Furthermore, based on the direction label, the camera image, the target ship image and the target berthing environment image can be fused.

[0087] In the embodiment of the present disclosure, the fusion of the camera image, the target ship image and the target berthing environment image is achieved through an image fusion model.

[0088] Therefore, as an optional implementation, the panoramic camera image fusion method for ship berthing also includes: obtaining a second training data set, the second training data set includes multiple second training samples, each second training sample includes: a sample camera image, a direction label corresponding to the sample camera image, a sample ship image, a direction label corresponding to the sample ship image, a sample berthing environment image, a direction label corresponding to the sample berthing environment image and a sample panoramic image, wherein different second training samples correspond to different sample panoramic images, and the sample camera image, the sample ship image and the sample berthing environment image are images obtained by splitting the sample panoramic image according to the corresponding direction labels; training the image fusion model to be trained according to the second training data set to obtain a pre-trained image fusion model.

[0089] In this embodiment, training data is configured, and the training data includes sample camera images, direction labels corresponding to the sample camera images, sample ship images, direction labels corresponding to the sample ship images, sample berthing environment images, direction labels corresponding to the sample berthing environment images, and sample panoramic images. The model is trained with the training data, and the model can synthesize images collected in multiple directions into a panoramic image based on the direction labels.

[0090] In addition, the training data can be obtained by first obtaining a sample panoramic image and then splitting the sample panoramic image according to the acquisition direction to obtain images of various types and corresponding image labels. The acquisition method is relatively simple.

[0091] Furthermore, by training the image fusion model, the pre-trained image fusion model performs panoramic image fusion according to the input relevant information to obtain the target panoramic image.

[0092] In the disclosed embodiment, the image fusion model may be a model with strong image processing capabilities, such as a deep learning model or a large model.

[0093] It can be seen that by combining the camera's captured images and the pre-configured image set and using the docking position information for image fusion, the requirements for the camera's image acquisition accuracy and the camera's setting method are reduced. In addition, without the need for complex image fusion technology, simple, effective and accurate panoramic image acquisition for docking can be achieved.

[0094] After the panoramic image is obtained, the panoramic image can be applied. Therefore, as an optional implementation, the panoramic camera image fusion method for ship berthing also includes: obtaining historical ship berthing data, the historical ship berthing data includes the berthing panoramic image and the actual ship berthing path corresponding to the berthing panoramic image; comparing the target panoramic image with the berthing panoramic image to obtain a comparison result; determining a recommended ship berthing path based on the comparison result and the historical ship berthing data; and displaying the target panoramic image and the recommended ship berthing path.

[0095] In some embodiments, after each ship berthing, a panoramic image for berthing and a corresponding actual ship berthing path are recorded, so that historical ship berthing data can be recorded in the ship system.

[0096] Then, the target panoramic image is compared with the docking panoramic image to obtain a comparison result, for example, by calculating image similarity to determine the docking panoramic image that has the highest similarity to the target panoramic image.

[0097] Further, based on the comparison result, a corresponding actual ship berthing path is determined from the historical ship berthing data, and the path is used as a recommended ship berthing path.

[0098] Thus, the target panoramic image and the recommended ship berthing path are displayed so that the ship driver can refer to them to perform the ship berthing operation.

[0099] Figure 6 FIG. 6 is a structural block diagram of a panoramic camera image fusion device 600 for ship berthing according to an exemplary embodiment. Figure 6 As shown, the device comprises:

[0100] The acquisition module 601 is configured to acquire the berthing position information of the ship, wherein the berthing position information includes the ship coordinates of the ship in a coordinate system with the berthing position as the coordinate origin and the direction of the ship relative to the berthing position.

[0101] The fusion module 602 is configured to obtain a camera image captured by a target camera of the ship according to the direction of the ship relative to the berthing position; determine a target ship image and a target berthing environment image according to the berthing position information and a pre-configured image set, wherein the pre-configured image set includes: a plurality of ship images and berthing environment images corresponding to the berthing position information respectively; and perform image fusion according to the camera image, the target ship image and the target berthing environment image to obtain a target panoramic image.

[0102] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0103] Figure 7 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include: a processor 701 , a memory 702 . The electronic device 700 may also include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0104] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned panoramic camera image fusion method for ship berthing. The memory 702 is used to store various types of data to support the operation of the electronic device 700, and these data may include, for example, instructions for any application or method used to operate on the electronic device 700, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 705 may include: Wi-Fi module, Bluetooth module, NFC module, etc.

[0105] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned XXXX method.

[0106] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned panoramic camera image fusion method for ship berthing are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions may be executed by the processor 701 of the electronic device 700 to complete the above-mentioned panoramic camera image fusion method for ship berthing.

[0107] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0108] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0109] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A panoramic camera image fusion method for ship berthing, characterized in that: include: Acquiring berthing position information of a ship, the berthing position information including the ship coordinates of the ship in a coordinate system with the berthing position as the coordinate origin and the direction of the ship relative to the berthing position; Acquiring a camera image captured by a target camera of the ship according to the direction of the ship relative to the berthing position; Determining a target ship image and a target berthing environment image according to the berthing position information and a pre-configured image set, wherein the pre-configured image set includes: a plurality of ship images and berthing environment images respectively corresponding to the berthing position information; Performing image fusion according to the camera image, the target ship image and the target berthing environment image to obtain a target panoramic image; The preconfigured image set includes a ship image set and a berthing environment image set, the ship image set includes a plurality of ship images corresponding to preset ship coordinates, the berthing environment image set includes a plurality of berthing environment images corresponding to preset ship coordinates, and determining the target ship image and the target berthing environment image according to the berthing position information and the preconfigured image set includes: determining a target ship image according to the ship images respectively corresponding to the plurality of preset ship coordinates, the ship coordinates, and the direction of the ship relative to the berthing position; Determining a target berthing environment image according to the berthing environment images respectively corresponding to the plurality of preset ship coordinates and the ship coordinates; The determining of the target ship image according to the ship images respectively corresponding to the plurality of preset ship coordinates, the ship coordinates and the direction of the ship compared to the berthing position comprises: Determine a first ship image according to the ship images respectively corresponding to the plurality of preset ship coordinates and the ship coordinates; determining a second ship image based on the first ship image, the direction of the ship relative to the berthing position, and a pre-trained image transformation model; The target ship image is determined according to the first ship image and the second ship image.

2. The method for fusion of panoramic camera images for ship berthing according to claim 1, characterized in that: The ship includes cameras arranged in multiple directions respectively, the direction of the ship relative to the berthing position is the direction of the bow of the ship relative to the berthing position, and acquiring the camera image captured by the target camera of the ship according to the orientation of the ship relative to the berthing position includes: Determine, from the multiple directions, a target direction having a smallest angle between the direction of the bow of the ship and the direction of the berthing position; Acquire a camera image captured by the target camera set in the target direction of the ship.

3. The method for fusion of panoramic camera images for ship berthing according to claim 1, characterized in that: The panoramic camera image fusion method for ship berthing also includes: Acquire a first training data set, the first training data set comprising a plurality of first training samples, each of the first training samples comprising: a sample ship image, a sample ship direction corresponding to the sample ship image, and a transformed image corresponding to the sample ship image, wherein similarities between sample ship images in different first training samples are less than a preset similarity, and / or sample ship directions corresponding to sample ship images in different first training samples are different; The image transformation model to be trained is trained according to the first training data set to obtain the pre-trained image transformation model, and the pre-trained image transformation model is used to output the second ship image according to the input first ship image and the direction of the ship compared to the berthing position.

4. The method for fusion of panoramic camera images for ship berthing according to claim 1, characterized in that: The ship coordinates include a first ship coordinate and a second ship coordinate, the ship positions corresponding to the first ship coordinate and the second ship coordinate are relative ship positions, and determining the target berthing environment image according to the berthing environment images respectively corresponding to the plurality of preset ship coordinates and the ship coordinates, includes: Determining a first berthing environment image corresponding to the first ship coordinates from the berthing environment images respectively corresponding to the plurality of preset ship coordinates; Determining a second berthing environment image corresponding to the second ship coordinates from the berthing environment images respectively corresponding to the plurality of preset ship coordinates; Deduplication processing is performed on the first docking environment image and the second docking environment image to obtain the target docking environment image.

5. The method for fusion of panoramic camera images for ship berthing according to claim 1, characterized in that: The performing image fusion according to the camera image, the target ship image and the target berthing environment image comprises: Determine the direction labels corresponding to the camera image, the target ship image and the target berthing environment image respectively; wherein the direction label corresponding to the camera image is a first direction, the direction label corresponding to the target ship image is a second direction, and the direction labels corresponding to the target berthing environment image are a third direction and a fourth direction; A pre-trained image fusion model is input according to the camera image, the target ship image, the target berthing environment image, the direction label corresponding to the camera image, the direction label corresponding to the target ship image and the direction label corresponding to the target berthing environment image to obtain the target panoramic image output by the pre-trained image fusion model.

6. The method for fusion of panoramic camera images for ship berthing according to claim 5, characterized in that: The panoramic camera image fusion method for ship berthing also includes: Acquire a second training data set, the second training data set comprising a plurality of second training samples, each second training sample comprising: a sample camera image, a direction label corresponding to the sample camera image, a sample ship image, a direction label corresponding to the sample ship image, a sample berthing environment image, a direction label corresponding to the sample berthing environment image, and a sample panoramic image, wherein different second training samples correspond to different sample panoramic images, and the sample camera image, the sample ship image, and the sample berthing environment image are images obtained by splitting the sample panoramic image according to the corresponding direction labels; The image fusion model to be trained is trained according to the second training data set to obtain the pre-trained image fusion model.

7. The method for fusion of panoramic camera images for ship berthing according to claim 1, characterized in that: The panoramic camera image fusion method for ship berthing also includes: Acquiring historical ship berthing data, wherein the historical ship berthing data includes a berthing panoramic image and an actual ship berthing path corresponding to the berthing panoramic image; Comparing the target panoramic image with the berthing panoramic image to obtain a comparison result; Determining a recommended ship berthing route according to the comparison result and the historical ship berthing data; The target panoramic image and the recommended ship berthing path are displayed.

8. A panoramic camera image fusion system for ship berthing, characterized in that: include: A plurality of cameras are respectively arranged in multiple directions of the ship for collecting camera images; A ship control system is connected to the multiple cameras respectively, and is used to implement the panoramic camera image fusion method for ship berthing as described in any one of claims 1 to 7.

Images