Port positioning method and device, and storage medium

Abstract

Embodiments of the present application disclose a port positioning method and device and a storage medium. The method comprises: acquiring a monitoring image, identifying the monitoring image by using a preset category detection model to determine a target object included in the monitoring image; identifying the monitoring image by using a preset coordinate detection model to determine a three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to a world coordinate system, and determining position information of the target object in the world coordinate system.

Description

TECHNICAL FIELD

[0001] The present application relates to the positioning field, and in particular to a port positioning method and device and a storage medium. BACKGROUND

[0002] Port transportation mainly relies on container trucks, and long-time heavy work will lead to low efficiency and safety hazards. The port operation area has the characteristics of relative closure, low speed and controllability, and is suitable for unmanned trucks, AGV (Automated Guided Vehicle) and other unmanned vehicles. However, at present, the port is generally in the process of intelligent upgrading, and transportation is still mainly driven by drivers, with a small number of unmanned vehicles as auxiliary.

[0003] In the driving process of the unmanned vehicle, the positioning information of the vehicle itself and other people and vehicle targets in the surrounding needs to be measured for planning the motion state and moving path. The vehicle self-positioning is mainly realized by measuring the latitude and longitude information by using the global positioning system (GPS), and can also be measured by using the roadside RFID, inertial navigation and other methods. The positioning of the surrounding unmanned vehicles is generally realized by a unified scheduling platform. The vehicle reports its position and motion state information to the platform, which uniformly processes and sends the information to other vehicles. In addition, the surrounding other people and vehicle target perception is mainly realized by loading laser radar, camera and other methods to model the environment and detect the target in real time, and measure the positioning information and motion state of the corresponding people and vehicle targets. In addition, all vehicles in the port area can be equipped with a positioning system to report their positioning information to the scheduling platform for unified processing.

[0004] In the port operation area, the vehicle-mounted sensor is easily disturbed, leading to positioning failure. For example, during driving, the surrounding containers are densely stacked, and large container trucks are shielded, resulting in a blind area of the laser radar and other sensors, and there is a risk of measurement omission. For another example, when the unmanned vehicle drives under the large crane such as the tower crane and the gantry crane, the GPS sensor signal is shielded by the metal structure, and the self-positioning performance is sharply decreased.

[0005] In addition, the scheme of positioning the surrounding other vehicle targets by the scheduling platform requires that the vehicles in the port area are uniformly equipped with positioning sensors and connected to the scheduling platform, which has very high construction and management costs, and cannot measure the positioning of pedestrians in the port area. SUMMARY

[0006] To solve the related technical problems, the embodiments of the present application provide a port positioning method, device and storage medium.

[0007] The technical scheme of the embodiments of the present application is realized as follows:

[0008] The embodiment of the present application provides a port positioning method, the method comprises:

[0009] An image is acquired, a preset category detection model is used to identify the image, and a target object included in the image is determined.

[0010] A preset coordinate detection model is used to identify the image, a three-dimensional frame coordinate of the target object is determined, the three-dimensional frame coordinate is converted to a world coordinate system, and position information of the target object in the world coordinate system is determined.

[0011] In the above scheme, the category detection model comprises a feature extraction unit, a category regression unit and a target category identification unit.

[0012] The image is identified by using the preset category detection model, and the target object included in the image is determined.

[0013] The features of the image are extracted by the feature extraction unit.

[0014] The features are identified by the category regression unit and the target category identification unit, and the target object included in the image is determined.

[0015] In the above scheme, the generation method of the category detection model comprises:

[0016] A first training sample set is acquired; the first training sample set comprises at least one sample image and a category label corresponding to each sample image; the sample image contains an object positioned in a port; and the category label represents the category of the object.

[0017] A first deep neural network obtained by training the first training sample set is used as the category detection model.

[0018] In the above scheme, the coordinate detection model comprises a feature extraction unit, a coordinate frame regression unit, a space constraint unit and a three-dimensional frame coordinate conversion unit.

[0019] The target object is identified by using the preset coordinate detection model, and the three-dimensional frame coordinate of the target object is determined.

[0020] The features of the target object in the image are extracted by the feature extraction unit.

[0021] The features of the target object are identified by the coordinate frame regression unit, and the three-dimensional frame coordinate of the target object corresponding to the features is obtained; the three-dimensional frame coordinate comprises image horizontal and vertical coordinate values of eight vertices of a cubic frame.

[0022] Correspondingly, the converting the stereoscopic frame coordinates into the world coordinate system to determine the position information of the target object in the world coordinate system comprises:

[0023] The stereoscopic frame coordinates are converted into coordinates in the world coordinate system by using the space constraint unit and the stereoscopic frame coordinate conversion unit, as the position information of the target object in the world coordinate system.

[0024] In the above scheme, the method for generating the coordinate detection model comprises:

[0025] A second training sample set is obtained; the second training sample set comprises at least one sample image and a position label corresponding to each sample image; the sample image contains an object positioned in a port; and the position label represents the stereoscopic frame coordinates and the world coordinate system coordinates of the object.

[0026] The second deep neural network obtained by training the second training sample set is used as the coordinate detection model.

[0027] In the above scheme, the feature extraction unit is configured to extract features in the sample image.

[0028] The coordinate frame regression unit is configured to determine the image horizontal and vertical coordinate values of the 8 vertices of the cubic frame in the sample image according to the extracted features.

[0029] The space constraint unit takes the parallelism between the top and bottom frames of the cubic frame and the horizontal ground and the perpendicularity between the side frame of the cubic frame and the horizontal ground as constraint conditions.

[0030] The stereoscopic frame coordinate conversion unit comprises a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit.

[0031] The monitoring camera internal and external parameter calibration unit comprises a camera projection matrix, which is configured to realize the conversion between the stereoscopic frame coordinates and the world coordinate system coordinates.

[0032] The image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinates according to the camera projection matrix and the stereoscopic frame coordinates.

[0033] The second deep neural network obtained by training the second training sample set is used as the coordinate detection model, which comprises:

[0034] The second deep neural network is trained according to the second training sample set under the constraint condition that the top and bottom frames of the cubic frame are parallel to the horizontal ground and the side frame of the cubic frame is perpendicular to the horizontal ground, to determine the camera projection matrix, the trained coordinate frame regression unit and the image coordinate and world coordinate conversion unit.

[0035] According to the determined camera projection matrix and the trained coordinate frame regression unit and image coordinate and world coordinate conversion unit, the coordinate detection model is determined.

[0036] The embodiment of the present application provides a port positioning device, the device comprises:

[0037] The first processing module is configured to acquire a monitoring image, identify the monitoring image by using a preset category detection model, and determine a target object included in the monitoring image.

[0038] The second processing module is configured to identify the monitoring image by using a preset coordinate detection model, determine a three-dimensional frame coordinate of the target object, convert the three-dimensional frame coordinate to a world coordinate system, and determine position information of the target object in the world coordinate system.

[0039] In the above scheme, the category detection model comprises a feature extraction unit, a category regression unit and a target category identification unit.

[0040] The first processing module is configured to extract features of the monitoring image by using the feature extraction unit.

[0041] The features are identified by using the category regression unit and the target category identification unit, and the target object included in the monitoring image is determined.

[0042] In the above scheme, the device further comprises a first preprocessing module configured to generate a category detection model.

[0043] The first preprocessing module is specifically configured to acquire a first training sample set; the first training sample set comprises at least one sample image and a category label corresponding to each sample image; the sample image contains an object positioned in a port; and the category label represents a category of the object.

[0044] A first deep neural network obtained by training the first training sample set is taken as the category detection model.

[0045] In the above scheme, the coordinate detection model comprises a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit and a three-dimensional frame coordinate conversion unit.

[0046] The second processing module is configured to extract features of the target object in the monitoring image by using the feature extraction unit.

[0047] The features of the target object are identified by using the coordinate frame regression unit, and three-dimensional frame coordinates of the target object corresponding to the features are obtained; the three-dimensional frame coordinates comprise image horizontal and vertical coordinate values of eight vertices of a cubic frame.

[0048] Correspondingly, the second processing module is further configured to convert the cuboid frame coordinates into coordinates in the world coordinate system as position information of the target object in the world coordinate system by using the space constraint unit and the cuboid frame coordinate conversion unit.

[0049] In the scheme, the device further includes a second preprocessing module configured to generate the coordinate detection model.

[0050] The second preprocessing module is specifically configured to obtain a second training sample set, the second training sample set including at least one sample image and a position label corresponding to each sample image, the sample image containing an object positioned in a port, and the position label representing cuboid frame coordinates and world coordinate system coordinates of the object.

[0051] The second deep neural network obtained by training the second training sample set is taken as the coordinate detection model.

[0052] In the scheme, the feature extraction unit is configured to extract features in the sample image.

[0053] The coordinate frame regression unit is configured to determine image horizontal and vertical coordinate values of 8 vertices of a cuboid frame in which the features are located in the sample image according to the extracted features.

[0054] The space constraint unit takes the parallelism between the top and bottom frames of the cuboid and the verticality between the side frames of the cuboid to the horizontal ground as constraint conditions.

[0055] The cuboid frame coordinate conversion unit includes a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit.

[0056] The monitoring camera internal and external parameter calibration unit includes a camera projection matrix, which is configured to realize conversion between the cuboid frame coordinates and the world coordinate system coordinates.

[0057] The image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinates according to the camera projection matrix and the cuboid frame coordinates.

[0058] The second preprocessing module is specifically configured to take the parallelism between the top and bottom frames of the cuboid and the verticality between the side frames of the cuboid to the horizontal ground as constraint conditions, train the second deep neural network according to the second training sample set, determine the camera projection matrix, and determine the trained coordinate frame regression unit and the image coordinate and world coordinate conversion unit.

[0059] The coordinate detection model is determined according to the determined camera projection matrix, the trained coordinate frame regression unit, and the image coordinate and world coordinate conversion unit.

[0060] The embodiment of the present application provides a port positioning device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the port positioning method of any one of the above when executing the program.

[0061] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the port positioning method of any one of the above when executed by a processor.

[0062] The embodiment of the present application provides a port positioning method, device and storage medium, and the method comprises the following steps: acquiring a monitoring image, identifying the monitoring image by using a preset category detection model, and determining a target object included in the monitoring image; identifying the monitoring image by using a preset coordinate detection model, determining a three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to a world coordinate system, and determining position information of the target object in the world coordinate system. In this way, the target to be monitored is determined by the category detection model, and the monitoring target is accurately positioned by the coordinate detection model, so that the positioning problem of non-unmanned vehicles and pedestrian targets is solved, and the unmanned vehicle can be assisted to position itself when the sensor signal is disturbed. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 A flowchart of a port positioning method provided by the embodiment of the present application is shown in the figure;

[0064] Figure 2 A flowchart of a target positioning implementation method based on video monitoring provided by the application embodiment of the present application is shown in the figure;

[0065] Figure 3 A structural diagram of a port positioning device provided by the embodiment of the present application is shown in the figure;

[0066] Figure 4 A structural diagram of another port positioning device provided by the embodiment of the present application is shown in the figure;

[0067] Figure 5 A structural diagram of still another port positioning device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0068] The present application will be further described in detail below with reference to the embodiments.

[0069] Figure 1 A flowchart of a port positioning method provided by the embodiment of the present application is shown in the figure; as shown in the figure, the method can be applied to a server or other intelligent electronic devices; the method comprises the following steps: Figure 1 ​

[0070] Step 101, obtaining a monitoring image, identifying the monitoring image by using a preset category detection model, and determining a target object included in the monitoring image;

[0071] Step 102, identifying the monitoring image by using a preset coordinate detection model, determining a three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to a world coordinate system, and determining position information of the target object in the world coordinate system.

[0072] In application, the server communicates with a monitoring device, and the monitoring device, such as a camera, a camera, or the like, can obtain a monitoring image.

[0073] The monitoring image is obtained, including that the server receives a monitoring image photographed and sent by the monitoring device.

[0074] In some embodiments, the category detection model includes a feature extraction unit, a category regression unit, and a target category identification unit.

[0075] The monitoring image is identified by using the preset category detection model, and the target object included in the monitoring image is determined, including:

[0076] The features of the monitoring image are extracted by the feature extraction unit;

[0077] The extracted features are identified by the category regression unit and the target category identification unit, and the target object included in the monitoring image is determined.

[0078] In some embodiments, the category detection model and the coordinate detection model can be two branches of one identification model, and the category detection model and the coordinate detection model can share one feature extraction unit, so as to ensure that the identified object category and the determined coordinate are for the same target.

[0079] The feature extraction unit is implemented by a deep neural network, and is composed of a plurality of cascaded convolution layers, activation layers, and pooling layers. The input monitoring image passes through the feature extraction network, and a group of feature maps with the same size can be output. That is, the category detection model includes a feature extraction unit, a category regression unit, and a target category identification unit; and the coordinate detection model includes a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit, and a three-dimensional frame coordinate conversion unit.

[0080] Here, a total deep neural network can be constructed in advance, specifically including two branches. The first branch is used to obtain a category detection model by training a first deep neural network; and the other branch is used to obtain a coordinate detection model by training a second deep neural network.

[0081] The method can be applied to port object positioning. When applied, the monitoring device is installed and fixed, and then communicates with a server. The server can obtain images of the monitoring device, and then obtains a category detection model and a coordinate detection model corresponding to each monitoring device by training a deep neural network.

[0082] In some embodiments, the method for generating the category detection model comprises:

[0083] obtaining a first training sample set, wherein the first training sample set comprises at least one sample image and a category label corresponding to each sample image, the sample image contains an object positioned in a port, and the category label represents a category of the object;

[0084] obtaining a first deep neural network by training the first training sample set, as the category detection model;

[0085] The category detection model comprises a feature extraction unit, a category regression unit, and a target category identification unit.

[0086] The first training sample set can comprise at least one sample image and a category label corresponding to each sample image. For example, the content of the sample image can be a pedestrian or a vehicle, and the corresponding category labels are “pedestrian” and “vehicle” respectively. The vehicle can further comprise a container, a general truck, a crane, etc.

[0087] The category regression unit and the coordinate frame regression unit are implemented by a deep neural network and are composed of a plurality of cascaded fully connected layers.

[0088] The network input is a feature map output by the feature extraction network in the feature extraction unit, and the output of the category regression unit is a two-dimensional matrix Y label =[C1,C2,…,C n ]’, wherein each row vector C i =[k1,k2,…k n ] represents a category result of a target object, and each element k in the vector represents a confidence of the corresponding category. The category corresponding to the maximum value is the category of the target object, such as “pedestrian” or “vehicle”.

[0089] In some embodiments, the coordinate detection model comprises a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit, and a stereo frame coordinate conversion unit.

[0090] The method for determining the stereo frame coordinates of the target object by using the preset coordinate detection model comprises:

[0091] extracting features of the target object in the monitoring image by the feature extraction unit;

[0092] The feature of the target object is identified by the coordinate frame regression unit, and the stereoscopic frame coordinates of the target object corresponding to the feature are obtained; the stereoscopic frame coordinates include image horizontal and vertical coordinate values of 8 vertices of a cubic frame;

[0093] Correspondingly, the conversion of the stereoscopic frame coordinates to the world coordinate system to determine the position information of the target object in the world coordinate system includes:

[0094] The stereoscopic frame coordinates are converted into coordinates in the world coordinate system by using the space constraint unit and the stereoscopic frame coordinate conversion unit, as the position information of the target object in the world coordinate system.

[0095] The coordinate frame regression unit is used to output a two-dimensional matrix Y loc =[B1,B2,…,B n ], wherein each row vector B i =[x 1u ,y 1u ,x 2u ,y 2u ,x 3u ,y 3u ,x 4u ,y 4u ,x 1d ,y 1d ,x 2d ,y 2d ,x 3d ,y 3d ,x 3d ,y 3d ] represents the stereoscopic frame coordinates of a target object, and the elements therein represent the image horizontal and vertical coordinate values of 8 vertices of a cubic frame; x represents the horizontal coordinate, y represents the vertical coordinate, subscript u represents the object top frame coordinate, and subscript d represents the object bottom frame coordinate.

[0096] The space constraint unit is used for target detection reasoning and training process.

[0097] The stereoscopic frame coordinate conversion unit is used to convert the stereoscopic frame coordinates B output by the coordinate frame regression unit to the world coordinate system coordinates according to the internal and external parameters of the monitoring camera, and to optimize the target stereoscopic frame coordinates through the space constraint relationship.

[0098] In some embodiments, the generation method of the coordinate detection model includes:

[0099] A second training sample set is obtained; the second training sample set includes at least one sample image and a position label corresponding to each sample image; the sample image contains an object positioned in a port; and the position label represents the stereoscopic frame coordinates and the world coordinate system coordinates of the object.

[0100] The second deep neural network trained by training the second training sample set is taken as the coordinate detection model.

[0101] Specifically, the coordinate detection model (second deep neural network) comprises a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit, and a stereo frame coordinate conversion unit.

[0102] The feature extraction unit can be shared with the feature extraction unit of the category detection model.

[0103] The coordinate frame regression unit is configured to determine the image horizontal and vertical coordinate values of the 8 vertices of the cubic frame in which the feature is located in the sample image according to the feature extracted by the feature extraction unit. The coordinate regression unit can be implemented by using One Stage target detection algorithm (of course, other algorithms such as two-stage target detection algorithm can also be used, which is not limited here). In application, the coordinate regression unit can be implemented by a neural network comprising a convolution layer and a full connection layer, the input of which is a feature map, and the output of which is N groups of 16-dimensional values, N representing N possible targets in the image, and the 16-dimensional values representing the xy coordinate values of the 8 vertices of each stereo frame.

[0104] The spatial constraint unit takes the constraint condition that the top and bottom frames of the cubic frame are parallel to the horizontal ground and the side frames of the cubic frame are perpendicular to the horizontal ground.

[0105] The stereo frame coordinate conversion unit comprises a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit.

[0106] The monitoring camera internal and external parameter calibration unit comprises a camera projection matrix, which is configured to realize the conversion between the stereo frame coordinate and the world coordinate system coordinate.

[0107] The image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinate according to the camera projection matrix and the stereo frame coordinate.

[0108] The second deep neural network trained by training the second training sample set is taken as the coordinate detection model, which comprises:

[0109] The second deep neural network is trained according to the second training sample set under the constraint condition that the top and bottom frames of the cubic frame are parallel to the horizontal ground and the side frames of the cubic frame are perpendicular to the horizontal ground, to determine the camera projection matrix, the trained coordinate frame regression unit, and the image coordinate and world coordinate conversion unit.

[0110] The coordinate detection model is determined according to the determined camera projection matrix, the trained coordinate frame regression unit, and the image coordinate and world coordinate conversion unit.

[0111] In particular, the camera projection matrix can be determined in advance by a training process. Here, let P represent the camera projection matrix, which is used to convert image coordinates B = [x, y] (i.e. B i = [x 1u ,y 1u ,x 2u ,y 2u ,x 3u ,y 3u ,x 4u ,y 4u ,x 1d ,y 1d ,x 2d ,y 2d ,x 3d ,y 3d ,x 3d ,y 3d ]) to world coordinate system coordinates [X, Y, Z]: [X, Y, Z] = P -1 [x, y, 1].

[0112] The cuboid frame of the world coordinate system needs to satisfy the spatial constraint relationship, i.e. the constraint condition of the spatial constraint unit is:

[0113] (1) The top frame and the bottom frame of the cuboid of the target object need to be parallel to the horizontal ground;

[0114] (2) The side frame of the cuboid of the target object needs to be perpendicular to the horizontal ground;

[0115] i.e. it needs to satisfy: Y 1u = Y 2u = Y 3u = Y 4u ,Y 1d = Y 2d = Y 3d = Y 4d ;

[0116] X 1u = X 1d ,X 2u = X 2d ,X 3u = X 3d ,X 4u = X 4d .

[0117] In the process of target detection reasoning, if multiple target objects are detected, the least squares method is applied to estimate the horizontal ground Y d , and it is taken as the bottom coordinate of the cuboid frame of all target objects;

[0118]

[0119] Where argmin represents the objective function Y at its minimum value d The value is given by n, which represents the total number of detected targets, i, which represents the i-th target, and j, which represents the j-th angle of the bottom coordinates.

[0120] During training, in addition to the coordinate regression loss SmoothL1-Loss, a spatial constraint loss Geometry-Loss is added:

[0121]

[0122] L total =L smothL +L gromerry ;

[0123] Where n represents the total number of detected targets, j represents the j-th angle of the bottom or top surface coordinates, k represents the k-th angle of the top or bottom surface coordinates, and Y... d,j The y-coordinate of the j-th corner of the bottom frame is represented by the coordinate Y. d,k The y-coordinate of the k-th corner of the bottom frame is represented by the coordinate Y. u,j The y-coordinate of the j-th corner of the top frame is represented by the coordinate Y. u,k X represents the ordinate of the k-th corner of the top frame. u,j X represents the x-coordinate of the j-th corner of the top frame. d,j L represents the x-coordinate of the j-th corner of the bottom frame. smothL L represents the coordinate regression loss function. gromerry L represents the spatial constraint loss function. total This represents the loss function used during training.

[0124] The 3D frame coordinate transformation unit includes: a monitoring camera intrinsic and extrinsic parameter calibration unit, an image coordinate to world coordinate conversion unit, etc.

[0125] During training, the camera projection matrix P = K[R|t] is determined by training the intrinsic and extrinsic parameter calibration unit of the monitoring camera, where K is the intrinsic parameter matrix, R is the extrinsic parameter rotation matrix, t is the translation matrix, and k is the k distortion parameter.

[0126]

[0127]

[0128] Here, f is the focal length of the surveillance camera, u0 and v0 are the coordinates of the optical center; t x t y t z These are the coordinates of the camera center relative to the origin in the x, y, and z directions, respectively. ω, ψ are respectively the pitch angle, the yaw angle and the roll angle of the camera rotating with the y axis as the main axis. The conversion relationship between the world coordinate system coordinates [X, Y, Z] and the image coordinates [x, y] is as follows:

[0129]

[0130] In the case of considering lens distortion, the distortion parameters k = [k1, k2, k3, p1, p2] satisfy:

[0131] x k =x(1+k1r 2 +k2r 4 +k3r 6 )+2p1xy+p2(r 2 +2x 2 )

[0132] y k =y(1+k1r 2 +k2r 4 +k3r 6 )+2p2xy+p1(r 2 +2y 2 )

[0133] wherein r = sqrt(x^2+y^2), sqrt represents square root; [k1, k2, k3] represents the radial distortion parameter, [p1, p2] represents the tangential distortion, [x k ,y k ] is the coordinate in the final image.

[0134] In the process of training the second deep neural network, the image coordinates [x, y] and the world coordinate system coordinates [X, Y, Z] are known parameters, which are respectively used as the sample stereo frame coordinates and the corresponding labels required for training.

[0135] [x k ,y k ], u0, v0 are known parameters, and u0, v0 are determined based on the actual monitoring device, i.e. the camera.

[0136] [k1, k2, k3], [p1, p2], f, ω, ψ, t x , t y , t z and other parameters are obtained by using the least square method; specifically, in the coordinate detection model (i.e. the second deep neural network), the feature extraction unit and the coordinate frame regression unit need to be trained, and the image of the second training sample set is used as the input, and the stereo frame coordinates [x, y] are used as the label.

[0137] Image coordinates [x, y] and world coordinate system coordinates [X, Y, Z] are used to solve P = argmin((PΔx k -Δx k ) 2 +(PΔy k -Δy k ) 2 ). The overall training process is: using the stereo frame coordinates and corresponding world coordinate system coordinates in the training sample set, solving P = argmin((PΔx k -Δx k ) 2 +(PΔy k -Δy k ) 2 ), calculating [k1, k2, k3], [p1, p2], f, ω, ψ, t x , t y , t z and other parameters; using the image and corresponding stereo frame coordinates in the training sample set, training the feature extraction and coordinate regression unit, and using the solved P to convert the stereo frame coordinates to the world coordinate system coordinates during training, introducing a spatial constraint unit to assist training and improve training accuracy. When applied, after inputting the monitoring image, the image coordinates can be obtained through the feature extraction unit and the coordinate frame regression unit, and the world coordinate system coordinates can be obtained through the coordinate conversion.

[0138] In actual application, for the port monitoring scene, the camera projection matrix parameters can also be simplified. The coordinate system is established with the camera center ground projection point as the coordinate origin and the camera optical axis as the coordinate x axis, so t x =t y =0, ω=0. In addition, according to the characteristics of the monitoring camera, the image center coordinates u0, v0 are known quantities, the xy direction focal length is equal f x =f y , the camera installation roll angle ψ=0, and the tangential distortion is ignored, i.e. p1=p2=0 (the above parameters are set based on actual conditions, such as camera horizontal installation and lens not rotating). The final projection matrix is simplified as:

[0139]

[0140] In view of the number of monitoring cameras and the difficulty of coordinate measurement, the solving method can be simplified in combination with the characteristics of the port area. A large number of containers are stacked in the port operation area, and the container size strictly meets the specification requirements, such as 20GP specification box length, width and height are 6.096×2.438×2.591 meters. In the model training process, 3 groups of vertices of 3 visible edges of the container with known size are selected to obtain the coordinate difference Δx k , Δyk , the actual size of the container set ΔX, ΔY, ΔZ can be solved by the equation set.

[0141] In particular, a plurality of sets of container vertices in different regions of the image can be selected to solve the projection matrix by an optimization method such as least squares to improve the solution accuracy.

[0142] P = argmin ((PΔx k - Δx k ) 2 +(PΔy k - Δy k ) 2 )

[0143] Finally, 2 or more points are selected in the image, the longitude and latitude coordinates in the world coordinate system are measured, the horizontal displacement t x ,t y and the yaw angle ω of the monitoring camera in the world coordinate system are calculated, and the image coordinate to world coordinate system longitude and latitude conversion matrix P world is calculated by the foregoing equation solving process.

[0144] That is, during the model training process, regarding the sample selection of the second training sample set, the sample and the corresponding label with known size, known stereo frame coordinates of the horizontal and vertical coordinates in the image, and known coordinate position in the world coordinate system can be selected to improve the accuracy of training and simplify the difficulty of training.

[0145] The image coordinate and world coordinate conversion unit calculates the longitude and latitude coordinates [X, Y, Z] of the vehicle and pedestrian target in the world coordinate system according to the stereo frame coordinates [x, y] of the target object obtained by the coordinate frame regression unit and the camera projection matrix P world determined by the monitoring camera internal and external parameter calibration unit: [X, Y, Z] = P -1 [x, y, 1].

[0146] The application embodiment provides a target positioning method based on video monitoring to solve the positioning problem of non-unmanned vehicles and pedestrian targets, and can assist the unmanned vehicle in positioning itself when the sensor signal is disturbed.

[0147] Figure 2 A flowchart of a target positioning implementation method based on video monitoring provided by the application embodiment is provided; the method is applied to a target positioning device, and the device comprises a target detection module and a coordinate conversion module; the method comprises:

[0148] After the target detection module obtains the monitoring image, the monitoring image is input into the target detection network to obtain the target category and the image coordinates (stereo frame coordinates);

[0149] The coordinate conversion module converts the coordinates of the acquired target in the image to the world coordinate system based on the camera's intrinsic and extrinsic parameters, and calculates the corresponding latitude and longitude information to achieve positioning.

[0150] like Figure 3 As shown, the target detection module includes: a feature extraction unit, a category regression unit, and a target category recognition unit;

[0151] The coordinate transformation module includes: a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit, and a 3D bounding box coordinate transformation unit;

[0152] The target detection module and the coordinate conversion module use the same feature extraction unit.

[0153] Each unit will be explained separately below.

[0154] The feature extraction unit is implemented using a deep neural network and consists of several cascaded convolutional layers, activation layers, and pooling layers. The input monitoring image passes through the feature extraction network, which outputs a set of feature maps of the same size.

[0155] The category regression unit and the bounding box regression unit are implemented using a deep neural network, consisting of several cascaded fully connected layers. The network input is the feature map output by the feature extraction network in the feature extraction unit, and the output of the category regression network is a two-dimensional matrix Y. label =[C1,C2,…,C n ]', where each row vector C i =[k1,k2,…k n The vector represents the category result of a target object. Each element k in the vector represents the confidence score of the corresponding category, and the category corresponding to the maximum value is the category of the target object (e.g., person, vehicle). The coordinate regression network outputs a two-dimensional matrix Y. loc =[B1,B2,…,B n ], where each row vector B i =[x 1u ,y 1u ,x 2u ,y 2u ,x 3u ,y 3u ,x 4u ,y 4u ,x 1d ,y 1d ,x 2d ,y 2d ,x 3d ,y 3d ,x 3d ,y 3drepresents a cuboid frame coordinate of a target object, wherein elements respectively represent image horizontal and vertical coordinate values of 8 vertices of the cuboid frame, subscript u represents a top surface frame coordinate of the target object, and subscript d represents a bottom surface frame coordinate of the target object.

[0156] The space constraint unit is configured to, in a target detection inference and training process, convert image coordinates B output by a coordinate regression network to world coordinate system coordinates according to internal and external parameters of a monitoring camera, and optimize the cuboid frame coordinates of the target object through a space constraint relationship.

[0157] Specifically, P represents a projection matrix of the monitoring camera, and image coordinates B = [x, y] are converted to world coordinate system coordinates [X, Y, Z] as follows: [X, Y, Z] = P -1 [x, y, 1].

[0158] The cuboid frame coordinates in the world coordinate system need to satisfy a space constraint relationship as follows:

[0159] (1) The cuboid top surface frame and the cuboid bottom surface frame of the target object need to be parallel to the horizontal ground.

[0160] (2) The cuboid side frames of the target object need to be perpendicular to the horizontal ground.

[0161] That is, the following conditions need to be satisfied: Y 1u =Y 2u =Y 3u =Y 4u ,Y 1d =Y 2d =Y 3d =Y 4d .

[0162] X 1u =X 1d ,X 2u =X 2d ,X 3u =X 3d ,X 4u =X 4d .

[0163] In the target detection inference process, if multiple target objects are detected, the least square method is applied to estimate the horizontal ground Y d , and the estimated value is taken as the bottom surface coordinate of the cuboid frame of all target objects.

[0164]

[0165] In the training process, a space constraint loss Geometry-Loss is added on the basis of a coordinate regression loss SmoothL1-Loss.

[0166]

[0167] L total =L smothL +L gromerry

[0168] The stereoscopic frame coordinate conversion unit can specifically include a unit for monitoring camera internal and external parameter calibration, image coordinate and world coordinate conversion, etc.

[0169] The internal and external parameter calibration unit needs to calculate the camera projection matrix P = K[R | t], i.e., the internal parameter matrix K, the external parameter rotation matrix R, the translation matrix t, and the distortion parameter k:

[0170] Here, f is the focal length of the monitoring camera, u0 and v0 are the coordinates of the optical center, t x , t y , and t z are the coordinates of the camera center in the xyz direction with respect to the coordinate origin, ω and ψ are the pitch angle and yaw angle and roll angle of the camera with the y-axis as the main axis. The conversion relationship between the world coordinate system coordinates [X, Y, Z] and the image coordinates [x, y] is:

[0171]

[0172] In the case of considering lens distortion, the distortion parameter k = [k1, k2, k3, p1, p2] satisfies:

[0173] x k = x (1 + k1r 2 + k2r 4 + k3r 6 ) + 2p1xy + p2(r 2 + 2x 2 )

[0174] y k = y (1 + k1r 2 + k2r 4 + k3r 6 ) + 2p2xy + p1(r 2 + 2y 2 )

[0175] Where [k1, k2, k3] represents the radial distortion parameter, [p1, p2] represents the tangential distortion, and [x k , y k ] are the coordinates in the final image.

[0176] In the port monitoring scene, the projection matrix parameters are simplified, the coordinate system is established with the camera center as the coordinate origin and the camera optical axis as the coordinate x-axis on the ground, and then t x = ty = 0, ω = 0. In addition, according to the monitoring camera characteristics, the image center coordinates u0, v0 are known quantities, and the xy direction focal length is equal f x = f y , the camera installation roll angle ψ = 0, and the tangential distortion is ignored, i.e., p1 = p2 = 0. Finally, the projection matrix is simplified as:

[0177]

[0178] In view of the number of monitoring cameras and the difficulty of coordinate measurement, it is necessary to simplify the solving method combined with the characteristics of the port area. A large number of containers are stacked in the port operation area, and the container size strictly meets the specification requirements, such as 20GP specification box length width height 6.096*2.438*2.591 meters. Selecting the visible 3 edges of the container, 3 groups of vertices are obtained, and the coordinate difference Δx k , Δy k of the container in the image is obtained, and the actual size ΔX, ΔY, ΔZ of the container is collected to solve the equation set. In particular, multiple groups of container vertices in different regions of the image can be selected to solve the projection matrix by using the least squares method and other optimization methods to improve the solving accuracy:

[0179] P = argmin ((PΔx k - Δx k ) 2 + (PΔy k - Δy k ) 2 )

[0180] Finally, 2 or more points in the image are selected, and the longitude and latitude coordinates in the world coordinate system are measured, and the horizontal displacement t x , t y and the yaw angle ω of the monitoring camera in the world coordinate system are calculated, and the image coordinate to world coordinate system longitude and latitude conversion matrix P world is calculated through the foregoing equation solving process.

[0181] The image coordinate and world coordinate conversion unit calculates the longitude and latitude coordinates [X, Y, Z] of the target in the world coordinate system according to the target stereo coordinates [x, y] obtained by the target detection module and the camera projection matrix P world solved in (1): [X, Y, Z] = P -1 [x, y, 1].

[0182] The method provided by the embodiment of the present application can realize target positioning of people and vehicles through video monitoring, can effectively assist the self-driving truck in judging the positions and driving states of surrounding targets, does not need to additionally install positioning devices and access platforms, and greatly reduces development and management costs. When the GPS sensor signals of the self-driving vehicle are shielded, the self-driving truck can be effectively assisted in self-positioning, and has better applicability.

[0183] Figure 4 A structural schematic diagram of a port positioning device provided by the embodiment of the present application is shown in FIG. 1. Figure 4 As shown in the figure, the device comprises:

[0184] A first processing module, configured to acquire a monitoring image, identify the monitoring image by using a preset category detection model, and determine a target object included in the monitoring image.

[0185] A second processing module, configured to identify the monitoring image by using a preset coordinate detection model, determine a three-dimensional frame coordinate of the target object, convert the three-dimensional frame coordinate to a world coordinate system, and determine position information of the target object in the world coordinate system.

[0186] Specifically, the category detection model comprises a feature extraction unit, a category regression unit and a target category identification unit.

[0187] The first processing module is configured to extract features of the monitoring image by using the feature extraction unit.

[0188] The features are identified by using the category regression unit and the target category identification unit, and the target object included in the monitoring image is determined.

[0189] Specifically, the device further comprises a first preprocessing module configured to generate a category detection model.

[0190] The first preprocessing module is specifically configured to acquire a first training sample set; the first training sample set comprises at least one sample image and a category label corresponding to each sample image; the sample image contains an object positioned in a port; and the category label represents a category of the object.

[0191] A first deep neural network obtained by training the first training sample set is taken as the category detection model.

[0192] Specifically, the coordinate detection model comprises a feature extraction unit, a coordinate frame regression unit, a space constraint unit and a three-dimensional frame coordinate conversion unit.

[0193] The second processing module is configured to extract features of the target object in the monitoring image by using the feature extraction unit.

[0194] The feature of the target object is identified by the coordinate frame regression unit, and a three-dimensional frame coordinate of the target object corresponding to the feature is obtained; the three-dimensional frame coordinate includes image horizontal and vertical coordinate values of eight vertices of a cubic frame;

[0195] Correspondingly, the second processing module is further configured to convert the three-dimensional frame coordinate into a coordinate in the world coordinate system by using the space constraint unit and the three-dimensional frame coordinate conversion unit, as position information of the target object in the world coordinate system.

[0196] Specifically, the device further includes a second preprocessing module configured to generate the coordinate detection model.

[0197] The second preprocessing module is specifically configured to obtain a second training sample set; the second training sample set includes at least one sample image and a position label corresponding to each sample image; the sample image contains an object positioned in a port; and the position label represents a three-dimensional frame coordinate and a world coordinate system coordinate of the object.

[0198] A second deep neural network obtained by training the second training sample set is taken as the coordinate detection model.

[0199] Specifically, the feature extraction unit is configured to extract a feature in the sample image.

[0200] The coordinate frame regression unit is configured to determine, according to the extracted feature, image horizontal and vertical coordinate values of eight vertices of a cubic frame in which the feature is located in the sample image.

[0201] The space constraint unit takes as constraint conditions that a top surface frame and a bottom surface frame of the cubic frame are parallel to a horizontal ground surface and a side surface frame of the cubic frame is perpendicular to the horizontal ground surface.

[0202] The three-dimensional frame coordinate conversion unit includes a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit.

[0203] The monitoring camera internal and external parameter calibration unit includes a camera projection matrix, which is configured to realize conversion between the three-dimensional frame coordinate and the world coordinate system coordinate.

[0204] The image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinate according to the camera projection matrix and the three-dimensional frame coordinate.

[0205] The second preprocessing module is specifically configured to train the second deep neural network according to the second training sample set, determine a camera projection matrix, and determine a trained coordinate frame regression unit and an image coordinate and world coordinate conversion unit, with the constraint condition that the top and bottom frames of the cube are parallel to the horizontal ground and the side frames of the cube are perpendicular to the horizontal ground.

[0206] The coordinate detection model is determined according to the determined camera projection matrix, the trained coordinate frame regression unit, and the image coordinate and world coordinate conversion unit.

[0207] It should be noted that the port positioning device provided in the above embodiments is only used as an example to illustrate the division of the above program modules when implementing the corresponding positioning method. In actual application, the above processing can be completed by different program modules according to needs, that is, the internal structure of the server is divided into different program modules to complete all or part of the above processing. In addition, the device and the corresponding method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0208] Figure 5 A structural schematic diagram of a port positioning device provided in an embodiment of the present application is shown in Figure 5 The device 50 includes a processor 501 and a memory 502 for storing a computer program capable of running on the processor; when the processor 501 runs the computer program, it performs the following: acquiring a monitoring image, identifying the monitoring image by using a preset category detection model to determine a target object included in the monitoring image; identifying the monitoring image by using a preset coordinate detection model to determine a three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to a world coordinate system, and determining position information of the target object in the world coordinate system.

[0209] The processor 501 can also perform the method shown in Figure 1 The port positioning method embodiment shown in Figure 1 The specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0210] In actual application, the device 50 can also include at least one network interface 503. The various components in the device 50 are coupled together through a bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between the components. The bus system 504 includes a data bus in addition to a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, only the data bus is shown in Figure 5The various buses are shown as bus system 504. The number of processors 501 can be at least one. The network interface 503 is used for the wired or wireless communication between the device 50 and other devices.

[0211] The memory 502 in the embodiments of the present application is used to store various types of data to support the operation of the device 50.

[0212] The method disclosed in the above embodiments of the present application can be applied in the processor 501 or implemented by the processor 501. The processor 501 can be an integrated circuit chip with a processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 501 or the instruction in the form of software. The above processor 501 can be a general processor, a digital signal processor (DSP, DiGital Signal Processor), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc. The processor 501 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the hardware decoding processor can be directly embodied to execute the above method, or the hardware and software modules in the decoding processor can be combined to execute the above method. The software module can be located in the storage medium, which is located in the memory 502. The processor 501 reads the information in the memory 502 and combines the hardware to complete the steps of the above method.

[0213] In the exemplary embodiments, the device 50 can be implemented by one or more application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic device (PLD, Programmable Logic Device), complex programmable logic device (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate Array), general processor, controller, microcontroller (MCU, Micro Controller Unit), microprocessor (Microprocessor), or other electronic elements, for executing the above method.

[0214] This invention also provides a computer-readable storage medium storing a computer program thereon; when the computer program is run by a processor, it performs the following: acquiring a monitoring image, identifying the monitoring image using a preset category detection model, and determining the target object included in the monitoring image; identifying the monitoring image using a preset coordinate detection model, determining the 3D bounding box coordinates of the target object, converting the 3D bounding box coordinates to a world coordinate system, and determining the position information of the target object in the world coordinate system.

[0215] Specifically, when the computer program is run by the processor, it can also execute tasks such as... Figure 1 The method shown is the same as Figure 1 The port positioning method embodiments shown belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0216] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0217] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0218] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0219] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, the foregoing program can be stored in a computer readable storage medium, and the program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0220] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if the integrated unit is realized in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product and stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods of the embodiments of the present application. The foregoing storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc and various storage medium capable of storing program codes.

[0221] It should be noted that "first", "second", and the like are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence.

[0222] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0223] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of port positioning, characterized in that, The method comprises: acquiring a monitoring image, identifying the monitoring image using a preset category detection model, and determining a target object included in the monitoring image; identifying the monitoring image using a preset coordinate detection model, determining a three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to a world coordinate system, and determining position information of the target object in the world coordinate system; the coordinate detection model comprises: a feature extraction unit, a coordinate frame regression unit, a spatial constraint unit, and a three-dimensional frame coordinate conversion unit; the feature extraction unit is configured to extract features in a sample image; the coordinate frame regression unit is configured to determine image horizontal and vertical coordinate values of eight vertices of a cubic frame in which the features are located in the sample image according to the extracted features; the spatial constraint unit takes the constraint condition that a top frame and a bottom frame of the cubic frame are parallel to a horizontal ground and a side frame of the cubic frame is perpendicular to the horizontal ground; the three-dimensional frame coordinate conversion unit comprises: a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit; the monitoring camera internal and external parameter calibration unit comprises a camera projection matrix, which is configured to realize conversion between a three-dimensional frame coordinate and a world coordinate system coordinate; the image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinate according to the camera projection matrix and the three-dimensional frame coordinate; wherein the operation of identifying the target object using the preset coordinate detection model, determining the three-dimensional frame coordinate of the target object, converting the three-dimensional frame coordinate to the world coordinate system, and determining the position information of the target object in the world coordinate system comprises: extracting features of the target object in the monitoring image by the feature extraction unit; identifying the features of the target object by the coordinate frame regression unit to obtain the three-dimensional frame coordinate of the target object corresponding to the features; the three-dimensional frame coordinate comprises image horizontal and vertical coordinate values of eight vertices of a cubic frame; and the three-dimensional frame coordinate is converted to a coordinate in the world coordinate system by the spatial constraint unit and the three-dimensional frame coordinate conversion unit, and is taken as the position information of the target object in the world coordinate system. The method for generating the coordinate detection model comprises: obtaining a second training sample set; the second training sample set comprises at least one sample image and a position label corresponding to each sample image; the sample image contains an object positioned in a port; the position label represents a cuboid frame coordinate and a world coordinate system coordinate of the object; a second deep neural network obtained by training the second training sample set is taken as the coordinate detection model; wherein, taking the cuboid as a constraint condition, wherein the top and bottom frames of the cuboid are parallel to the horizontal ground and the side frames of the cuboid are perpendicular to the horizontal ground, the second deep neural network is trained according to the second training sample set to determine a camera projection matrix, a trained coordinate frame regression unit and an image coordinate and world coordinate conversion unit; and the coordinate detection model is determined according to the determined camera projection matrix, the trained coordinate frame regression unit and the image coordinate and world coordinate conversion unit.

2. The method of claim 1, wherein, The category detection model comprises: a feature extraction unit, a category regression unit and a target category identification unit. The method for identifying the target object included in the monitoring image by using the preset category detection model comprises: The feature extraction unit is used to extract the features of the monitoring image; The category regression unit and the target category identification unit are used to identify the extracted features to determine the target object included in the monitoring image.

3. The method according to claim 1 or 2, characterized in that, The method for generating the category detection model comprises: A first training sample set is obtained; the first training sample set comprises at least one sample image and a category label corresponding to each sample image; the sample image contains an object positioned in a port; the category label represents the category of the object; A first deep neural network obtained by training the first training sample set is taken as the category detection model.

4. A port positioning device, characterized in that The device comprises: The first processing module is configured to obtain a monitoring image, identify the monitoring image by using a preset category detection model, and determine a target object included in the monitoring image; The second processing module is configured to identify the monitoring image by using a preset coordinate detection model, determine a three-dimensional frame coordinate of the target object, convert the three-dimensional frame coordinate to a world coordinate system, and determine position information of the target object in the world coordinate system; the coordinate detection model comprises: a feature extraction unit, a coordinate frame regression unit, a space constraint unit, and a three-dimensional frame coordinate conversion unit; the feature extraction unit is configured to extract features in a sample image; the coordinate frame regression unit is configured to determine image horizontal and vertical coordinate values of eight vertices of a cubic frame in which the features are located in the sample image according to the extracted features; the space constraint unit takes as constraint conditions that a top frame and a bottom frame of the cubic frame are parallel to a horizontal ground and a side frame of the cubic frame is perpendicular to the horizontal ground; the three-dimensional frame coordinate conversion unit comprises: a monitoring camera internal and external parameter calibration unit and an image coordinate and world coordinate conversion unit; the monitoring camera internal and external parameter calibration unit comprises a camera projection matrix, which is configured to convert the three-dimensional frame coordinate and the world coordinate system coordinate; the image coordinate and world coordinate conversion unit is configured to determine the world coordinate system coordinate according to the camera projection matrix and the three-dimensional frame coordinate; The second processing module is further configured to extract features of the target object in the monitoring image by using the feature extraction unit, identify the features of the target object by using the coordinate frame regression unit, and obtain three-dimensional frame coordinates of the target object corresponding to the features; the three-dimensional frame coordinates comprise image horizontal and vertical coordinate values of eight vertices of a cubic frame; the second processing module is further configured to convert the three-dimensional frame coordinates to coordinates in the world coordinate system by using the space constraint unit and the three-dimensional frame coordinate conversion unit, and use the coordinates in the world coordinate system as the position information of the target object in the world coordinate system. The device further comprises a second preprocessing module configured to generate the coordinate detection model; the second preprocessing module is further configured to obtain a second training sample set; the second training sample set comprises at least one sample image and a position label corresponding to each sample image; the sample image contains an object positioned in a port; the position label represents three-dimensional frame coordinates and world coordinate system coordinates of the object; the second preprocessing module is further configured to obtain a second deep neural network trained by the second training sample set as the coordinate detection model; the second preprocessing module is further configured to take as constraint conditions that a top frame and a bottom frame of the cubic frame are parallel to a horizontal ground and a side frame of the cubic frame is perpendicular to the horizontal ground, train the second deep neural network according to the second training sample set, determine a camera projection matrix, and train the coordinate frame regression unit and the image coordinate and world coordinate conversion unit, and determine the coordinate detection model according to the determined camera projection matrix, the trained coordinate frame regression unit, and the image coordinate and world coordinate conversion unit.

5. The apparatus of claim 4, wherein, The category detection model comprises: a feature extraction unit, a category regression unit, and a target category identification unit; The first processing module is configured to extract features of the monitoring image by using the feature extraction unit. The feature extraction unit, the category regression unit and the target category identification unit are configured to identify the extracted feature and determine whether the target object is included in the monitoring image.

6. The apparatus of claim 4 or 5, wherein, The device further comprises a first preprocessing module configured to generate a category detection model. The first preprocessing module is specifically configured to obtain a first training sample set, wherein the first training sample set comprises at least one sample image and a category label corresponding to each sample image, the sample image contains an object positioned in a port, and the category label represents a category of the object. A first deep neural network obtained by training the first training sample set is taken as the category detection model.

7. A port positioning device, characterized in that Comprise: A processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute the steps of the method of any one of claims 1 to 3 when running the computer program.

8. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3.

Images