Scrap steel image acquisition method, device and system, electronic equipment and medium

Abstract

The scrap steel image acquisition method comprises the steps that in the process that each batch of scrap steel is transferred to an unloading area, a sub-area in the unloading area where the batch of scrap steel is unloaded by a steel grabbing device and a photographing site corresponding to the sub-area are identified, and the photographing site represents the position where the batch of scrap steel is unloaded by the steel grabbing device in the sub-area; in response to determining that the steel grabbing device leaves the unloading area, obtaining three-dimensional point cloud data about the sub-area; based on the three-dimensional point cloud data, the magnification times of sub-region photographing are determined; and the image of the batch of scrap steel is collected according to the photographing site and the magnification times. Also provided is a scrap steel image acquisition system, comprising: an image acquisition device for acquiring an image comprising at least a part of the unloading area; the three-dimensional scanning device is used for scanning at least one part of the unloading area to obtain three-dimensional point cloud data of the corresponding area; and the processing device is operably connected with the image acquisition device and the three-dimensional scanning device and is used for executing the scrap steel image acquisition method.

Description

Technical Field

[0001] The present disclosure relates to the field of machine vision, and in particular to a method, device, scrap steel image acquisition system, electronic equipment, computer-readable storage medium, and computer program product for acquiring scrap steel images. Background Art

[0002] With the development requirements of the green economy, in order to promote the early realization of the goals of carbon peak and carbon neutrality, and to reduce dependence on iron ore, many steel mills at home and abroad use scrap steel as an important raw material for steelmaking. Steel mills need to purchase a large amount of scrap steel every year, and the number of types of scrap steel is huge, the material types are complex, and the shapes and sizes are various. Many of them are attached with impurities such as soil residue and oil stains. Traditionally, steel mills have mostly graded the acquired scrap steel through manual visual quality inspection. After the scrap steel collection, storage and transportation are completed, the rating results are manually entered and uploaded to the ERP or MES system. Summary of the invention

[0003] The present disclosure provides a method, an apparatus, a scrap steel image acquisition system, an electronic device, a computer-readable storage medium, and a computer program product for acquiring scrap steel images.

[0004] According to one aspect of the present disclosure, a method for capturing images of scrap steel is provided, comprising: during a process in which a steel grasping device transfers a batch of scrap steel to a scrap steel unloading area each time, identifying a sub-area within the scrap steel unloading area into which the batch of scrap steel is unloaded by the steel grasping device and a corresponding photographing position of the sub-area, the photographing position indicating a position within the sub-area where the steel grasping device unloads the batch of scrap steel; in response to determining that the steel grasping device leaves the scrap steel unloading area, acquiring three-dimensional point cloud data about the sub-area; determining a magnification for photographing the sub-area based on the three-dimensional point cloud data; and initiating capturing images of the batch of scrap steel according to the photographing position and the magnification.

[0005] According to another aspect of the present disclosure, a device for capturing images of scrap steel is provided, comprising: a first module for identifying a sub-area within the scrap steel unloading area into which the batch of scrap steel is unloaded by the steel grasping device and a corresponding photographing position within the sub-area during each process in which the steel grasping device transfers a batch of scrap steel to the scrap steel unloading area, the photographing position indicating the position within the sub-area where the steel grasping device unloads the batch of scrap steel; a second module for acquiring three-dimensional point cloud data about the sub-area in response to determining that the steel grasping device leaves the scrap steel unloading area; a third module for determining a magnification for photographing the sub-area based on the three-dimensional point cloud data; and a fourth module for initiating capturing images of the batch of scrap steel according to the photographing position and the magnification.

[0006] According to another aspect of the present disclosure, a scrap steel image acquisition system is provided, comprising: an image acquisition device for acquiring images including at least a portion of a scrap steel unloading area; a three-dimensional scanning device for scanning at least a portion of the scrap steel unloading area to obtain three-dimensional point cloud data of the corresponding area; and a processing device operably connected to the image acquisition device and the three-dimensional scanning device, for executing the method for acquiring scrap steel images of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method for capturing scrap steel images of the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to enable a computer to execute the method for acquiring scrap steel images of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method for acquiring scrap steel images of the present disclosure when executed by a processor.

[0010] According to one or more embodiments of the present disclosure, not only can the unloading position of the steel grabbing device be accurately located, while ensuring that each batch of scrap steel being unloaded is photographed in a covered manner and overlapping of the photographing areas is avoided, but the magnification factor can also be automatically adjusted for the corresponding photographing area to ensure that the sizes of each batch of scrap steel photographed are relatively uniform, thereby improving the accuracy of subsequent scrap steel grading.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0014] Figure 2An exemplary flow chart of a method for acquiring scrap steel images according to an embodiment of the present disclosure is shown;

[0015] Figure 3 An exemplary flow chart showing additional steps of a method for acquiring scrap steel images according to an embodiment of the present disclosure;

[0016] Figure 4 An exemplary flow chart showing additional steps of a method for acquiring scrap steel images according to an embodiment of the present disclosure;

[0017] Figure 5 An exemplary flow chart showing additional steps of a method for acquiring scrap steel images according to an embodiment of the present disclosure;

[0018] Figure 6 An exemplary flow chart showing additional steps of a method for acquiring scrap steel images according to an embodiment of the present disclosure;

[0019] Figure 7 An exemplary flow chart showing additional steps of a method for acquiring scrap steel images according to an embodiment of the present disclosure;

[0020] Figure 8 A diagram showing an exemplary scrap steel image acquisition scenario according to an embodiment of the present disclosure is shown.

[0021] Fig. 9 A structural block diagram of a device for collecting scrap steel images according to an embodiment of the present disclosure is shown;

[0022] Fig.10 A structural block diagram of a scrap steel image acquisition system according to an embodiment of the present disclosure is shown;

[0023] Fig.11 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0027] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 is a schematic diagram illustrating an example system 100 in which the various methods described herein may be implemented, according to an example embodiment.

[0029] refer to Figure 1 The system 100 includes a client device 110 , a server 120 , and a network 130 that communicatively couples the client device 110 and the server 120 .

[0030] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 can be an application that needs to be downloaded and installed before running or a small program (lite app) as a lightweight application. In the case where the client application 112 is an application that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case where the client application 112 is a small program, the user 102 can directly run the client application 112 on the client device 110 by searching for the client application 112 in the host application (for example, by the name of the client application 112, etc.) or scanning the graphic code of the client application 112 (for example, a bar code, a QR code, etc.), without installing the client application 112. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (for example, a head-mounted device such as a smart watch, smart glasses, etc.) or other types of mobile devices. In some embodiments, the client device 110 can be a fixed computer device, such as a desktop, a server computer or other types of fixed computer devices.

[0031] The server 120 is typically a server deployed by an Internet Service Provider (ISP) or an Internet Content Provider (ICP). The server 120 may represent a single server, a cluster of multiple servers, a distributed system, or a cloud server that provides basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). It will be understood that although Figure 1 The server 120 is shown in FIG. 1 to communicate with only one client device 110 , but the server 120 may provide background services for multiple client devices simultaneously.

[0032] Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. Network 130 can be a wired or wireless network. In some embodiments, the data exchanged through network 130 is processed using technologies and / or formats including hypertext markup language (HTML), extensible markup language (XML), etc. In addition, encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet protocol security (IPsec) can also be used to encrypt all or some links. In some embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0033] For the purpose of the embodiments of this disclosure, Figure 1 In the example of , the client application 112 may be a scrap steel image acquisition application, which may provide various functions related to the scrap steel image acquisition process, such as the division of the scrap steel unloading area, the selection of reference photographing locations, the parameter setting of the image acquisition device and / or the three-dimensional scanning device, etc. Correspondingly, the server 120 may be a server used together with the above-mentioned scrap steel image acquisition application. The server 120 may provide online scrap steel image acquisition services to the client application 112 running in the client device 110. Alternatively, the client application 112 running in the client device 110 may provide local scrap steel image acquisition services.

[0034] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0035] In the related art, the method used by steel mills to grade the scrap steel they purchase is mostly manual visual quality inspection. After the scrap steel is collected, stored and transported, the rating results are manually entered and uploaded to the ERP or MES system. However, manual visual quality inspection has the following disadvantages: 1) During the manual visual quality inspection and rating process of scrap steel, the rating personnel need to climb onto the cargo ship or transport vehicle loaded with scrap steel, which has blind spots for observation and makes it difficult to visually inspect the entire scrap steel unloading area, which also poses a major safety hazard; 2) The rating process is completely dependent on the subjective experience of the rating personnel, which is prone to misjudgment, and thus prone to disputes with scrap steel suppliers, and lacks means for subsequent verification; 3) Manual rating cannot be automatic around the clock, and the operation process takes a long time, affecting the efficiency of scrap steel transportation.

[0036] With the continuous advancement of technology, some scrap steel rating methods based on machine vision have also begun to enter the field of scrap steel rating. However, the areas captured by these scrap steel rating methods based on machine vision have large overlaps, and each scrap steel image varies in size, affecting the accuracy of scrap steel rating for the entire ship or vehicle.

[0037] To this end, the present invention proposes a more efficient and versatile scrap steel image acquisition method and system based on machine vision, which can replace the traditional scrap steel grading method of manual visual quality inspection, realize uninterrupted intelligent collection of scrap steel images indoors and outdoors around the clock, and improve the efficiency and accuracy of scrap steel grading.

[0038] Figure 2 FIG. 2 shows an exemplary flow chart of a method 200 for collecting scrap steel images according to an embodiment of the present disclosure. Figure 1 The client or server implementation shown in Figure 2 Method 200 in.

[0039] like Figure 2 As shown, according to an embodiment of the present disclosure, a method 200 for collecting scrap steel images is provided, comprising the following steps 210 to 240.

[0040] In some embodiments, steps S210 to S240 may be performed each time the steel grabbing device transfers a batch of scrap steel to the scrap steel unloading area.

[0041] Step S210, identifying a sub-area in the scrap steel unloading area where a batch of scrap steel is unloaded by the steel grabbing device and a corresponding photographing position in the sub-area, the photographing position indicating the position in the sub-area where the steel grabbing device unloads the batch of scrap steel.

[0042] In the example, the steel grasping device may be a steel grasping machine or an electromagnetic suction cup, etc., and the present disclosure does not impose any limitation on this.

[0043] In an example, the scrap steel unloading area may be a deck of a cargo ship that can load scrap steel or a carriage of a transfer vehicle, etc., and the present disclosure does not impose any limitation on this.

[0044] As used in the present disclosure, the term "a batch of scrap steel" may refer to a collection of scrap steel that is transferred at one time by a steel grabbing device from a scrap steel transporter to a scrap steel recycler (for example, from a cargo ship loaded with scrap steel docked at a dock to a scrap steel unloading area on the dock (such as a scrap steel transport truck) or vice versa, etc.).

[0045] As used in the present disclosure, the term "photographing site" may refer to a focal point within the image to be captured monitored by a viewfinder (e.g., a hardware viewfinder or a software viewfinder) of an image acquisition device, that is, an imaging focal point within the image to be captured where a lens of the image acquisition device focuses or the clearest convergence point within the image to be captured. It is understood that the photographing site may not necessarily be located at the geometric center of the image to be captured, and the number of photographing sites within the image to be captured may be greater than or equal to one, and the present disclosure does not impose any limitation on this.

[0046] Step S220, in response to determining that the steel grasping device leaves the scrap steel unloading area, acquiring three-dimensional point cloud data about the sub-area.

[0047] It can be understood that in order to improve the transfer efficiency so that the weight of each batch of scrap steel grabbed is as much as possible (for example, thereby reducing the workload of the steel grabbing device in transporting the scrap steel back and forth), the volume of the steel grabbing device is usually large (for example, occupying multiple sub-areas within the scrap steel unloading area). Therefore, after the steel grabbing device leaves the scrap steel unloading area, obtaining three-dimensional point cloud data about the sub-area where a batch of scrap steel is unloaded can avoid the acquired three-dimensional point cloud data from containing three-dimensional point cloud information about the steel grabbing device, thereby effectively reducing or even eliminating the noise in the three-dimensional point cloud data of the sub-area.

[0048] As used in the present disclosure, the term "three-dimensional point cloud data" may refer to a collection of points in the form of spatial coordinates of each sampling point on the surface of an object obtained by scanning the object with the aid of a three-dimensional scanning device (such as a laser radar, etc.). The three-dimensional point cloud data may include not only coordinate information, but also intensity information and / or angle information, etc., and the present disclosure does not impose any limitation on this.

[0049] In an example, three-dimensional point cloud data about the sub-area may be acquired by a three-dimensional scanning device.

[0050] In the example, two-dimensional image data about the sub-area (e.g., the image of the sub-area taken) can be acquired by an image acquisition device, and then three-dimensional point cloud data can be generated based on the two-dimensional image data by Neural Radiance Fields (NeRF) technology to save costs. The present disclosure does not impose any restrictions on the method of acquiring three-dimensional point cloud data.

[0051] Step S230, determining the magnification factor for photographing the sub-area based on the three-dimensional point cloud data.

[0052] In practice, batches of scrap steel are usually unloaded into a scrap steel unloading area in a certain arrangement.

[0053] As an example but not a limitation, initially, the scrap steel unloading area is empty and does not include any batch of transferred scrap steel. Assuming that the scrap steel unloading area is divided into N non-overlapping sub-areas, wherein the N sub-areas completely cover the area occupied by the scrap steel unloading area, the earliest N batches of scrap steel are respectively unloaded to the corresponding N sub-areas, and the N batches of scrap steel form the first layer of scrap steel piles in the scrap steel unloading area. Subsequently, the N+1 to 2N batches of scrap steel are also respectively unloaded to the corresponding N sub-areas, and the N+1 to 2N batches of scrap steel form the second layer of scrap steel piles in the scrap steel unloading area, and so on.

[0054] Therefore, the three-dimensional point cloud data obtained about the sub-area can be used as characteristic information to characterize a batch of scrap steel corresponding to the sub-area, and the volume, weight, height, distance (for example, the distance between the unloading position of the batch of scrap steel and a fixed point such as an image acquisition device) and other information about the batch of scrap steel corresponding to the sub-area can be directly or indirectly derived from such three-dimensional point cloud data.

[0055] In the example, the magnification for photographing the sub-area corresponding to the batch of scrap steel can be determined based on the distance between the discharge position of the batch of scrap steel contained in the three-dimensional point cloud data and a fixed point such as an image acquisition device. It can be understood that the greater the distance between the discharge position of the scrap steel and the fixed point such as the image acquisition device, the greater the magnification required for photographing the corresponding sub-area. Of course, the present disclosure does not impose any restrictions on the relationship (e.g., a functional relationship, etc.) between the distance between the discharge position of the scrap steel and the fixed point such as the image acquisition device and the magnification required for photographing the corresponding sub-area.

[0056] In the example, under the condition that the height of a given layer of scrap steel pile does not vary much in various places, the three-dimensional point cloud data obtained about the sub-area can be the three-dimensional point cloud data about the entire scrap steel pile where a batch of scrap steel corresponding to the sub-area is located. Therefore, for each sub-area in a layer of scrap steel pile, the three-dimensional point cloud data of the layer of scrap steel pile can be obtained as the three-dimensional point cloud data about each sub-area, so as to reduce the workload of the three-dimensional scanning device and improve efficiency.

[0057] For example, continuing with the non-limiting example of the scrap steel pile mentioned above, assuming that the height of a given layer of scrap steel pile does not vary much in various places, then for at most N sub-regions in this layer of scrap steel pile, three-dimensional point cloud data of this layer of scrap steel pile can be obtained as the three-dimensional point cloud data of each of the at most N sub-regions, and it can be concluded from such three-dimensional point cloud data that the height information of the corresponding photographing locations of each of the at most N sub-regions can be consistent.

[0058] Step S240, initiating the acquisition of images of the batch of scrap steel according to the photographing position and magnification.

[0059] In the example, initiating the acquisition of images of a batch of scrap steel according to the photographing position and the magnification may include: sending an instruction to the image acquisition device so that the image acquisition device focuses on the corresponding sub-area where the scrap steel is located based on the photographing position included in the instruction, and photographs the sub-area according to the magnification included in the instruction to obtain an image, that is, an image of the batch of scrap steel corresponding to the sub-area.

[0060] In the related art, there are many deficiencies in the scrap steel rating method based on machine vision, such as large overlap of the photographed areas, and the size of each scrap steel image fluctuating, which will affect the accuracy of scrap steel rating of the entire ship or vehicle. With the help of method 200, not only can the position where the steel grabbing device unloads a batch of scrap steel be accurately located, thereby ensuring that each batch of scrap steel unloaded is covered and photographed, while avoiding the overlap of the photographed areas, but also the magnification of the photographing can be automatically adjusted for the corresponding photographing area, ensuring that the size of each batch of scrap steel images taken is relatively uniform, thereby improving the accuracy of subsequent scrap steel rating.

[0061] Figure 3 FIG. 1 is an exemplary flow chart showing additional steps of a method for collecting scrap steel images according to an embodiment of the present disclosure. Figure 3 As shown, before the steel grabbing device transfers each batch of scrap steel to the scrap steel unloading area, the method 200 further includes steps S201 to S202.

[0062] Step S201, acquiring an initial image including a scrap steel unloading area.

[0063] In an example, the initial image may be obtained from a data file storing previously acquired images including a scrap discharge area.

[0064] By way of example and not limitation, in a case where the scrap unloading area is the cabin of a transfer vehicle (e.g., a truck) that can be loaded with scrap steel, such a data file may include previously captured overhead or near-overhead images of the cabins of various types of transfer vehicles (e.g., the beds of trucks).

[0065] In another example, before the steel grabbing device grabs the first batch of scrap steel to be transferred, the image acquisition device may acquire an initial image including the scrap steel unloading area.

[0066] As used in the present disclosure, when referring to an image including a scrap steel unloading area, it means that the image contains the scrap steel unloading area, so that the image of the scrap steel unloading area can be obtained by a cropping operation or the like.

[0067] Step S202: dividing the scrap steel unloading area based on the initial image to obtain a plurality of sub-areas of the scrap steel unloading area.

[0068] In the example, the scrap steel unloading area can be divided based on the initial image according to factors such as the average volume of each batch of scrap steel to be transferred grabbed by the steel grabbing device, the working mode of the steel grabbing device boom, the length-to-width ratio of the scrap steel unloading area, etc.

[0069] As an example but not a limitation, the scrap steel unloading area may be divided into a plurality of sub-areas in a single row (especially when the length and width of the scrap steel unloading area are relatively large) or a plurality of sub-areas in a grid shape (for example, when the length and width of the scrap steel unloading area are not much different), etc. Of course, the division result may also be that the scrap steel unloading area only includes one sub-area, for example, each batch of scrap steel to be transferred grasped by the steel grasping device reaches an average level of occupying the area of ​​the entire scrap steel unloading area or the area of ​​the scrap steel unloading area itself is so small that it is not suitable to be divided into two or more sub-areas, etc. The present disclosure does not impose any limitation on the division method of the scrap steel unloading area.

[0070] Figure 4 FIG. 1 is an exemplary flow chart showing additional steps of a method for collecting scrap steel images according to an embodiment of the present disclosure. Figure 4 As shown, step S210 of method 200, i.e., identifying a sub-area in the scrap steel unloading area into which a batch of scrap steel is unloaded by the steel grabbing device and the corresponding photographing position of the sub-area, may include steps S211 to S214.

[0071] Step S211, obtaining a video stream recording the process of the steel grabbing device transferring a batch of scrap steel to the scrap steel unloading area.

[0072] In an example, the video stream may adopt a mainstream video file format to facilitate subsequent parsing and processing of the video stream.

[0073] In another example, a video stream may also refer to a sequence of images collected continuously in time, and the images in the sequence may adopt a mainstream image file format to facilitate subsequent image processing of the images. The present disclosure does not impose any limitation on the form of the video stream.

[0074] Step S212, determining the motion trajectory of the steel grasping device based on the video stream.

[0075] In an example, the motion trajectory of the steel grasping device may be determined by identifying the position coordinates of the steel grasping device from the video stream.

[0076] As an example and not limitation, the image acquisition device may be a gun-ball integrated camera device. Thus, for example, the motion trajectory of the steel gripping device may be determined (e.g., calculated) from video frames or images captured using the global field of view of a gun camera in the gun-ball integrated camera device by a deep learning algorithm.

[0077] In the example, the motion trajectory of the steel grabbing device may include a transfer trajectory of the steel grabbing device grabbing a batch of scrap steel from the scrap steel transporter (for example, the deck of a cargo ship loaded with scrap steel docked at the dock) and transferring and unloading the batch of scrap steel to a scrap steel unloading area at a scrap steel recycling party, and a motion trajectory of the steel grabbing device leaving the scrap steel unloading area and returning to the scrap steel transporter after unloading a batch of scrap steel.

[0078] Step S213, based on the movement trajectory of the steel grasping device, determining a sub-area within the scrap steel unloading area where the batch of scrap steel is unloaded by the steel grasping device.

[0079] In an example, a sub-region where a singular point (eg, a turning point or a position indicating that the steel grabbing device stays for a longer time) identified from the motion trajectory of the steel grabbing device is located can be determined as the sub-region into which the batch of scrap steel is discharged.

[0080] Step S214, using deep learning technology to determine the position where the steel grabbing device unloads the batch of scrap steel as the photo taking location.

[0081] It should be noted that the deep learning technology itself is known to ordinary technicians in the field, and any suitable type of deep learning technology can be used to determine the photo-taking location. The present disclosure does not impose any limitations on this and will not be elaborated here.

[0082] As mentioned above, the photographing position refers to, for example, the imaging focal point where the lens of the image acquisition device is focused in the image to be photographed or the clearest convergence point in the image to be photographed. Therefore, the photographing position determined by using deep learning technology can make the photographed image better focus on the distribution center point of the batch of scrap steel while covering the sub-area where the batch of scrap steel is located, avoiding unfavorable situations such as out-of-focus of the scrap steel in the photographed image or focusing on the edge part of the batch of scrap steel (for example, the scrap steel part with less mass or volume).

[0083] Figure 5 FIG. 1 is an exemplary flow chart showing additional steps of a method for collecting scrap steel images according to an embodiment of the present disclosure. Figure 5 As shown, step S220 of method 200, that is, in response to determining that the steel grasping device leaves the scrap steel unloading area, acquiring three-dimensional point cloud data about the sub-area may include steps S221 to S222.

[0084] Step S221, based on at least one of the video stream and the motion trajectory of the steel grabbing device, determining that the steel grabbing device leaves the scrap steel unloading area.

[0085] In an example, it may be determined based on the video stream acquired as in step S211 that the steel grabbing device leaves the scrap steel unloading area.

[0086] In another example, it may be determined that the steel grabbing device leaves the scrap steel unloading area based on the motion trajectory of the steel grabbing device as determined in step S212 .

[0087] In yet another example, it may be determined that the steel grabbing device leaves the scrap steel unloading area based on both the video stream and the motion trajectory of the steel grabbing device.

[0088] Step S222: Initiate scanning of the sub-area using a three-dimensional scanning device to obtain three-dimensional point cloud data of the sub-area.

[0089] As mentioned above, scanning the sub-area using the three-dimensional scanning device may include scanning only the corresponding sub-area or scanning the entire scrap steel pile layer including the corresponding sub-area, and the present disclosure does not impose any limitation on this.

[0090] Therefore, by obtaining three-dimensional point cloud data about the sub-area where the scrap steel is unloaded after the steel grabbing device leaves the scrap steel unloading area, it is possible to avoid the obtained three-dimensional point cloud data containing three-dimensional point cloud information about the steel grabbing device, thereby effectively reducing or even eliminating the noise in the three-dimensional point cloud data of the sub-area, and improving the accuracy of the obtained three-dimensional point cloud data.

[0091] Figure 6 FIG. 1 is an exemplary flow chart showing additional steps of a method for collecting scrap steel images according to an embodiment of the present disclosure. Figure 6As shown, step S230 of method 200, ie, determining the magnification for photographing the sub-region based on the three-dimensional point cloud data, may include steps S231 to S234.

[0092] Step S231, using the three-dimensional point cloud data to reconstruct the surface of a batch of scrap steel in the sub-area to obtain a surface mesh of the batch of scrap steel.

[0093] It should be noted that using the three-dimensional point cloud data of an object to reconstruct the surface of an object is known to ordinary technicians in the field. Any suitable type of three-dimensional point cloud processing method can be used to determine the surface mesh of a batch of scrap steel. The present disclosure does not impose any restrictions on this and will not be elaborated here.

[0094] In an example, the average height of the top surface of the batch of scrap steel may be calculated based on the obtained surface mesh of the batch of scrap steel.

[0095] Step S232, based on the surface grid of the batch of scrap steel and the ground point cloud data included in the three-dimensional point cloud data, determine the height of the top surface of the batch of scrap steel from the ground of the sub-area.

[0096] As used in this disclosure, the term "ground" may refer to any suitable given reference plane and is not limited to the ground plane.

[0097] As an example but not limitation, the ground of the sub-area may refer to the bottom surface of the scrap steel unloading area. Taking the truck bed as an example, the ground may refer to the bottom inner surface or the bottom outer surface of the truck bed.

[0098] It can be understood that when the ground of a given sub-area is given, the distance between the detection port (such as lens, ray receiving port, etc.) of an electronic device such as an image acquisition device and / or a three-dimensional scanning device and the ground can be measurable.

[0099] It can also be understood that the distance between the detection port of electronic equipment such as an image acquisition device and / or a three-dimensional scanning device and the ground is generally greater than the distance between the top surface of any batch of scrap steel and the ground (i.e., the height of the top surface of the batch of scrap steel from the ground of the sub-area). The reason is that the detection port of the electronic equipment needs to be able to acquire images and / or perform ray scanning on the sub-area from a top-down or near-top-down perspective.

[0100] In an example, the average height of the ground of the sub-area may be calculated based on the ground point cloud data included in the three-dimensional point cloud data.

[0101] Step S233, based on the position parameters of the image acquisition device for photographing the sub-area relative to the scrap steel unloading area and the height of the top surface of the batch of scrap steel from the ground of the sub-area, determine the distance between the image acquisition device and the photographing location.

[0102] In the example, the position parameters of the image acquisition device relative to the scrap steel unloading area may include the distance H between the lens of the image acquisition device and the ground of the sub-area (for example, the bottom surface of the scrap steel unloading area) (i.e., the height of the image acquisition device from the ground of the sub-area), and the distance d from the photographing position to the perpendicular line drawn by the image acquisition device and the ground.

[0103] As an example but not a limitation, let the height of the top surface of a given batch of scrap steel from the ground of the sub-area where the batch of scrap steel is located be denoted as h, then the distance s between the image acquisition device and the photographing location can be determined as

[0104] Step S234, determining the magnification for photographing the sub-area based at least on the distance between the image acquisition device and the photographing location.

[0105] In an example, the magnification at which the sub-region is to be photographed may be determined by making the magnification a function of the distance between the image acquisition device and the photographing site.

[0106] As mentioned above, the greater the distance between the discharge position of the scrap steel and a fixed point such as an image acquisition device, the greater the magnification factor required for photographing the corresponding sub-area. Thus, for example, a function formula can be constructed in which the photographing magnification factor increases with the increase of the distance between the image acquisition device and the photographing position, so as to select a magnification factor suitable for photographing the corresponding sub-area.

[0107] Figure 7 FIG. 1 is an exemplary flow chart showing additional steps of a method for collecting scrap steel images according to an embodiment of the present disclosure. Figure 7 As shown, step S234, that is, determining the magnification for photographing the sub-area based at least on the distance between the image acquisition device and the photographing site, may include steps S2341 to 2343.

[0108] Step S2341, determining a reference photographing location within the scrap steel unloading area and a reference distance between the image acquisition device and the reference photographing location.

[0109] In an example, the reference photographing location may be a point in a scrap unloading area (eg, a cargo box of a transfer truck) that is not loaded with any batch of transferred scrap. In other words, the reference photographing location may be determined before each batch of scrap is transferred.

[0110] As an example but not a limitation, in the case where the bottom inner surface of the truck bed is the ground, the reference photographing position may be a point on the bottom inner surface of the truck bed, and the distance between the lens of the acquisition device and the ground is H0, and the distance from the reference photographing position to the perpendicular line drawn through the image acquisition device and the ground is d0, then the reference distance s0 between the image acquisition device and the reference photographing position may be determined as s0=sqrt(H0 2 +d0 2 ).

[0111] It is understandable that the reference photographing location can be selected by a person skilled in the art as needed, and the present disclosure does not impose any limitation on this.

[0112] Step S2342, by making the image acquisition device focus on the reference photographing position and adjusting the magnification of the image acquisition device, the image acquisition device can just capture the entire area of ​​the scrap steel unloading area, and obtain the reference magnification of the image acquisition device.

[0113] It can be understood that the image acquisition device is able to capture the entire area of ​​the scrap steel unloading area, which in some cases can be equivalent to the boundary of the image taken by the image acquisition device basically coinciding with the perimeter of the scrap steel unloading area (for example, the frame of the truck bed), thereby obtaining a reference magnification k0.

[0114] Step S2343, based on the distance between the image acquisition device and the photographing site, the reference distance and the reference magnification, determine the magnification for photographing the sub-area.

[0115] As an example and not limitation, the magnification k at which the sub-region is to be photographed may be determined as

[0116]

[0117] It can be understood that although the above describes the determination of the magnification k for photographing a sub-area in the form of a linear function, a person skilled in the art can select any suitable function as needed to determine the magnification of the image acquisition device for photographing different sub-areas, and the present disclosure does not impose any limitations on this.

[0118] In some embodiments, initiating the acquisition of an image of a batch of scrap steel according to a photographing location and a magnification may include: making an image acquisition device focus on the photographing location and adjusting the magnification of the image acquisition device to the magnification, photographing the sub-area to obtain an image of the batch of scrap steel.

[0119] In this way, the magnification of the shooting can be automatically adjusted for the corresponding shooting area to ensure that the size of each batch of scrap steel images taken is relatively uniform, thereby improving the accuracy of subsequent scrap steel rating.

[0120] Figure 8 A diagram showing an exemplary scrap steel image acquisition scenario according to an embodiment of the present disclosure is shown.

[0121] It should be noted that although Figure 8 The scrap steel image collection scenario is illustrated using the scenario of crane unloading at a dock, but a person of ordinary skill in the art can understand that the scrap steel image collection method disclosed herein can be applied to any other suitable scrap steel transfer scenario, and the present disclosure does not impose any limitation on this.

[0122] In such Figure 8 In the scrap steel image acquisition scene shown, there are a scrap steel transfer device 1, a pole support device 2, an image acquisition device 3, a terminal card swiping device 4, an LED display device 5, an image processing device 6, a cabin 7, a transfer carriage 8, a gun-ball integrated camera 9, a fill light 10, a port crane base 11 and a steel grabber 12.

[0123] As shown in the figure, the scrap steel transfer device 1 can be deployed at the dockside, responsible for grabbing (or sucking) the scrap steel in the cabin 7 into the transfer car 8. The pole support device 2 can be deployed next to the parking space to facilitate the camera to face the scrap steel transfer vehicle as much as possible. The image acquisition device 3 (which includes a gun-ball integrated camera 9, a fill light 10, etc.) and the three-dimensional scanning device 13 can be set on the top of the pole support device 2. In addition, the terminal card swiping device 4 can be deployed at the bottom of the pole support device 2, and the LED display device 5 can be set in the middle of the pole support device 2. The image processing device 6 can be remotely deployed in the server room and connected to the image acquisition device 3, the terminal card swiping device 4, the LED display device 5 and / or the three-dimensional scanning device 13 through a communication line (for example, a wired communication line or a wireless communication channel or a combination thereof).

[0124] As shown in the figure, the pole bracket device 2 can be in an inverted "L" shape, and the top crossbeam has a mounting bracket for the gun-ball integrated camera 9 in the image acquisition device 3, a mounting bracket for the fill light 10, etc. The pole bracket device 2 can adopt a hollow design so that all communication lines and power supply lines are connected from the inside of the bracket, thereby avoiding the lines being exposed in a complex environment, and improving safety and durability.

[0125] As further shown in the figure, the scrap steel transfer device 1 can be composed of a port crane base 11, a steel grabber 12, etc. The operator controls the steel grabber 12 to transfer each batch of scrap steel according to the instructions prompted on the LED display device 5, so as to grab the scrap steel in the cabin 7 and unload it into the transfer carriage 8 in sequence.

[0126] It can be understood that the gun camera in the integrated gun-ball camera 9 can be responsible for the global vision control of the port crane unloading area, and the ball camera can be responsible for the photo collection of the scrap steel pictures in the transfer carriage 8. The fill light 10 can provide the lighting needs for night operations, and can also reduce the impact of the shadow of the sun obliquely shining in the transfer carriage 8 on the imaging of the integrated gun-ball camera 9 during the day, thereby ensuring the imaging quality of the scrap steel pictures.

[0127] As further shown in the figure, the terminal card swiping device 4 can be set at the bottom of the pole bracket device 2, and can be communicatively connected with other components through the communication line inside the pole bracket, so that the operator (such as the driver of the transfer vehicle) can initiate the intelligent collection and automatic grading process of scrap steel images.

[0128] Optionally, the LED display device 5 can be set in the middle of the pole support device 2, which can be communicatively connected with other components through the communication line inside the pole support to display character information in multiple colors (for example, displaying prompts for the operations that the operator of the scrap steel transfer device 1 needs to perform). Different information can be divided according to categories so as to be displayed in different colors.

[0129] The image processing device 6, as the core processing unit of the intelligent collection and automatic grading process of scrap steel images, may include an image processing server, which exchanges information and controls various components at the terminal site through communication lines.

[0130] For example, the image processing device 6 can automatically start the relevant task process by receiving the logistics code information, scrap steel image intelligent collection and automatic rating process start instructions transmitted by the terminal card swiping device 4. After all task processes are completed, the overall rating result of the transported scrap steel on the vehicle can be displayed on the interface of the terminal card swiping device 4 for the operator to view.

[0131] Optionally, the interactive information such as logistics coding information, intelligent collection of scrap steel images, and start instructions for the automatic grading process transmitted by the terminal card swiping device 4 can be alternatively implemented using a handheld terminal and supporting applications.

[0132] Optionally, when the factory's existing MES or ERP system interface can directly obtain interactive information such as logistics coding information that needs to be transmitted, intelligent collection of scrap steel images, and start instructions for the automatic grading process, the terminal card swiping device 4 may not be set up.

[0133] Optionally, before transferring each batch of scrap steel to the transfer carriage 8, an image recognition algorithm may be used to determine whether the parking position of the carriage 8 of the transfer vehicle is within the optimal shooting range of the image acquisition device 3. If not, the parking of the carriage 8 may be guided. For example, if the parking position of the vehicle does not meet the requirements, a prompt message may be generated and automatically fed back to the on-site LED display device 5 to prompt the operator (e.g., the driver) that the parking position of the vehicle does not meet the requirements and that the vehicle needs to be re-adjusted to enter the warehouse by backing until the vehicle is determined to be parked near the optimal position.

[0134] Optionally, the collected photos of each batch of scrap steel can be transmitted back to the image processing device 6 for storage.

[0135] Accordingly, the computer vision algorithm in the image processing device 6 can identify the collected scrap steel images from multiple feature dimensions such as thickness, material type, size, diameter, density fitting, non-steel impurities and dangerous seals, and comprehensively derive the evaluation level of the entire vehicle of transported scrap steel. In addition, after the image processing device 6 detects the presence of dangerous seals in the scrap steel image, the warning character information can be transmitted to the LED display device 5 via the communication line to promptly remind the operator of the scrap steel transport device 1 to take corresponding measures.

[0136] Fig. 9 A structural block diagram of a device for collecting scrap steel images according to an embodiment of the present disclosure is shown.

[0137] like Fig. 9 As shown, according to an embodiment of the present disclosure, a device 900 for capturing images of scrap steel is provided, comprising a first module 910, for identifying a sub-area within a scrap steel unloading area into which a batch of scrap steel is unloaded by a steel grabbing device and a corresponding photographing position of the sub-area, the photographing position indicating the position where the steel grabbing device unloads the batch of scrap steel within the sub-area; a second module 920, for acquiring three-dimensional point cloud data about the sub-area in response to determining that the steel grabbing device leaves the scrap steel unloading area; a third module 930, for determining a magnification for photographing the sub-area based on the three-dimensional point cloud data; and a fourth module 940, for initiating the capture of images of the batch of scrap steel according to the photographing position and the magnification.

[0138] Here, the operations of the above modules 910 to 940 of the device 900 are similar to the operations of steps 210 to 240 of the method 200 described above, and are not repeated here. In some embodiments, the device 900 may also include one or more additional modules or units to perform functions corresponding to the additional steps of the method 200 (for example, steps 201 to 202, steps 211 to 214, steps 221 to 222, steps 231 to 234, steps S2341 to 2343, etc.), which are not shown in the figure for the purpose of simplicity. Figure 7 middle.

[0139] In the related art, there are many deficiencies in the scrap steel rating method based on machine vision, such as large overlap of the photographed areas, and the size of each scrap steel image fluctuating, which will affect the accuracy of scrap steel rating of the entire ship or vehicle. With the aid of the device 900, not only can the position where the steel grabbing device unloads a batch of scrap steel be accurately located, thereby ensuring that each batch of scrap steel unloaded is covered and photographed, while avoiding the overlap of the photographed areas, but also the magnification of the photographing can be automatically adjusted for the corresponding photographing area, ensuring that the size of each batch of scrap steel images taken is relatively uniform, thereby improving the accuracy of subsequent scrap steel rating.

[0140] In some embodiments, the device 900 may also include a fifth module for obtaining an initial image including the scrap steel unloading area before the steel grabbing device transfers each batch of scrap steel to the scrap steel unloading area; and dividing the scrap steel unloading area based on the initial image to obtain multiple sub-areas of the scrap steel unloading area.

[0141] In some embodiments, the first module 910 of the device 900 may include: a first unit, used to obtain a video stream recording the process of a steel grasping device transferring a batch of scrap steel to a scrap steel unloading area; a second unit, used to determine the movement trajectory of the steel grasping device based on the video stream; a third unit, used to determine a sub-area within the scrap steel unloading area where the batch of scrap steel is unloaded by the steel grasping device based on the movement trajectory of the steel grasping device; and a fourth unit, used to use deep learning technology to determine the position where the steel grasping device unloads the batch of scrap steel as a photo-taking location.

[0142] In some embodiments, the second module 920 of the device 900 may include: a fifth unit, used to determine that the steel grabbing device has left the scrap steel unloading area based on at least one of the video stream and the motion trajectory of the steel grabbing device; and a sixth unit, used to initiate scanning of the sub-area using a three-dimensional scanning device to obtain three-dimensional point cloud data of the sub-area.

[0143] In some embodiments, the third module 930 of the device 900 may include: a seventh unit, used to reconstruct the surface of a batch of scrap steel in the sub-area using three-dimensional point cloud data to obtain a surface mesh of the batch of scrap steel; an eighth unit, used to determine the height of the top surface of the batch of scrap steel from the ground of the sub-area based on the surface mesh of the batch of scrap steel and the ground point cloud data included in the three-dimensional point cloud data; a ninth unit, used to determine the distance between the image acquisition device and the photographing location based on the position parameters of the image acquisition device to be photographed of the sub-area relative to the scrap steel unloading area and the height of the top surface of the batch of scrap steel from the ground of the sub-area; a tenth unit, used to determine the magnification of the sub-area to be photographed based at least on the distance between the image acquisition device and the photographing location.

[0144] In some embodiments, the tenth unit can be further configured to: determine a reference photographing location within the scrap steel unloading area and a reference distance between the image acquisition device and the reference photographing location; by making the image acquisition device focus on the reference photographing location and adjusting the magnification of the image acquisition device, so that the image acquisition device can just capture the entire area of ​​the scrap steel unloading area, and obtain the reference magnification of the image acquisition device; based on the distance between the image acquisition device and the photographing location, the reference distance and the reference magnification, determine the magnification for photographing the sub-area.

[0145] In some embodiments, the fourth module 940 of the device 900 can be further configured to: focus the image acquisition device on the photographing position and adjust the magnification of the image acquisition device to the magnification, photograph the sub-area to obtain an image of the batch of scrap steel.

[0146] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules, and / or at least some functions of multiple modules may be combined into a single module. The specific module discussed herein performing an action includes the specific module itself performing the action, or alternatively the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, the specific module that performs an action may include the specific module itself that performs the action and / or another module that the specific module calls or otherwise accesses to perform the action.

[0147] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Fig. 9The modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuits. Hardware logic / circuits can include integrated circuit chips (which include processors (e.g., central processing units (CPU), microcontrollers, microprocessors, digital signal processors (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0148] Fig.10 FIG. 2 shows a structural block diagram of a scrap steel image acquisition system according to an embodiment of the present disclosure. Fig.10 As shown, the scrap steel image acquisition system 1000 may include an image acquisition device 1010, used to acquire images including at least a portion of the scrap steel unloading area; a three-dimensional scanning device 1020, used to scan at least a portion of the scrap steel unloading area to obtain three-dimensional point cloud data of the corresponding area; and a processing device 1030, which is operably connected to the image acquisition device 1010 and the three-dimensional scanning device 1020 to execute the method for acquiring scrap steel images disclosed in the present invention, for example, the additional steps of method 200 and its various embodiments.

[0149] It is understood that the image acquisition device 1010 may include various suitable types of cameras, video cameras, etc., and the present disclosure does not impose any restrictions on this. The three-dimensional scanning device 1020 may include suitable types of commercially available three-dimensional scanning equipment such as radars and laser radars, and the present disclosure does not impose any restrictions on this. The processing device 1030 may be any customized or commercially available processor, such as a central processing unit (CPU), a distributed processing unit, a semiconductor-based microprocessor (for example, in the form of a microchip or a chipset), etc., and the present disclosure does not impose any restrictions on this.

[0150] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0151] According to an embodiment of the present disclosure, an electronic device is provided. The electronic device may include: at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for collecting scrap steel images of the present disclosure.

[0152] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions can be used to enable a computer to execute the method for collecting scrap steel images of the present disclosure.

[0153] According to an embodiment of the present disclosure, a computer program product is provided, which may include a computer program, and when the computer program is executed by a processor, the method for collecting scrap steel images of the present disclosure can be implemented.

[0154] It can be understood that the scrap steel image acquisition scheme proposed in the present disclosure has the advantages of versatility, automation, and no human intervention. With the scrap steel image acquisition scheme disclosed in the present disclosure, the unloading location can be accurately located, and while ensuring that more scrap steel is photographed in a covered manner, the probability of repeated photographing of the same place is effectively reduced, thereby significantly improving the coverage rate of photographing and reducing the repetition rate of photographing areas, so that the rating of the whole vehicle scrap steel can be more accurate. In addition, the scrap steel image acquisition scheme disclosed in the present disclosure can adaptively adjust the magnification of the photographing, ensure the size of each scrap steel image taken is stable, and improve the recognition accuracy.

[0155] refer to Fig.11 , a block diagram of an electronic device 1100 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0156] like Fig.11 As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0157] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 can be any type of device that can input information to the electronic device 1100. The input unit 1106 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1107 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 can include but is not limited to a disk and an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device and / or the like.

[0158] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as method 200 and its various embodiments. For example, in some embodiments, the method 200 and its various embodiments may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the method 200 described above and the additional steps of its various embodiments may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured in any other appropriate manner (eg, by means of firmware) to execute additional steps of the method 200 and its embodiments.

[0159] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0160] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0161] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0163] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0164] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0165] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0166] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for collecting scrap steel images, comprising: During the process of the steel grabbing device transferring a batch of scrap steel to the scrap steel unloading area, Identify a sub-area within the scrap steel unloading area where the batch of scrap steel is unloaded by the steel grabbing device and a corresponding photographing position within the sub-area, wherein the photographing position indicates a position within the sub-area where the steel grabbing device unloads the batch of scrap steel; In response to determining that the steel grabbing device leaves the scrap steel unloading area, acquiring three-dimensional point cloud data about the sub-area; Determining a magnification for photographing the sub-area based on the three-dimensional point cloud data; as well as Initiate collecting images of the batch of scrap steel according to the photographing position and the magnification.

2. The method according to claim 1, wherein: Before the steel grabbing device transfers each batch of scrap steel to the scrap steel unloading area, the method further includes: Acquiring an initial image including the scrap steel unloading area; The scrap steel unloading area is divided based on the initial image to obtain a plurality of sub-areas of the scrap steel unloading area.

3. The method according to claim 1 or 2, wherein: The method for identifying a sub-area in the scrap steel unloading area where the batch of scrap steel is unloaded by the steel grabbing device and a corresponding photographing position of the sub-area includes: Acquire a video stream recording a process in which the steel grabbing device transfers the batch of scrap steel to the scrap steel unloading area; determining a motion trajectory of the steel grasping device based on the video stream; Based on the motion trajectory of the steel grabbing device, determining the sub-area within the scrap steel unloading area into which the batch of scrap steel is unloaded by the steel grabbing device; The deep learning technology is used to determine the position where the steel grabbing device unloads the batch of scrap steel, which serves as the photo-taking location.

4. The method according to claim 3, wherein: In response to determining that the steel grabbing device leaves the scrap steel unloading area, acquiring three-dimensional point cloud data about the sub-area includes: Determining that the steel grabbing device leaves the scrap steel unloading area based on at least one of the video stream and the motion trajectory of the steel grabbing device; A three-dimensional scanning device is initiated to scan the sub-area to obtain three-dimensional point cloud data of the sub-area.

5. The method according to claim 4, wherein: Determining the magnification of photographing the sub-area based on the three-dimensional point cloud data includes: Reconstructing the surface of the batch of scrap steel in the sub-area by using the three-dimensional point cloud data to obtain a surface mesh of the batch of scrap steel; Determining the height of the top surface of the batch of scrap steel from the ground of the sub-area based on the surface grid of the batch of scrap steel and the ground point cloud data included in the three-dimensional point cloud data; Determine the distance between the image acquisition device and the photographing location based on the position parameters of the image acquisition device for photographing the sub-area relative to the scrap steel unloading area and the height of the top surface of the batch of scrap steel from the ground of the sub-area; The magnification factor for photographing the sub-area is determined based at least on the distance between the image acquisition device and the photographing site.

6. The method according to claim 5, wherein: Determining the magnification for photographing the sub-area based at least on the distance between the image acquisition device and the photographing site includes: Determine a reference photographing position in the scrap steel unloading area and a reference distance between the image acquisition device and the reference photographing position; By making the image acquisition device focus on the reference photographing position and adjusting the magnification of the image acquisition device, so that the image acquisition device can just capture the entire area of ​​the scrap steel unloading area, a reference magnification of the image acquisition device is obtained; The magnification for photographing the sub-area is determined based on the distance between the image acquisition device and the photographing site, the reference distance, and the reference magnification.

7. The method according to claim 1 or 2, wherein: Initiating the acquisition of images of the batch of scrap steel according to the photographing position and the magnification comprises: The image acquisition device is made to focus on the photographing position and the magnification of the image acquisition device is adjusted to the magnification, and the sub-area is photographed to obtain the image of the batch of scrap steel.

8. A device for collecting scrap steel images, comprising: The first module is used for identifying a sub-area in the scrap steel unloading area where the batch of scrap steel is unloaded by the steel grabbing device and a corresponding photographing position of the sub-area during each process of the steel grabbing device transferring a batch of scrap steel to the scrap steel unloading area, wherein the photographing position indicates a position in the sub-area where the steel grabbing device unloads the batch of scrap steel; A second module is configured to acquire three-dimensional point cloud data about the sub-area in response to determining that the steel grabbing device leaves the scrap steel unloading area; A third module is used to determine the magnification of taking a photo of the sub-area based on the three-dimensional point cloud data; as well as The fourth module is used to initiate the acquisition of images of the batch of scrap steel according to the photographing position and the magnification.

9. A scrap steel image acquisition system, comprising: An image acquisition device for acquiring an image including at least a portion of a scrap steel unloading area; A three-dimensional scanning device is used to scan at least a part of the scrap steel unloading area to obtain three-dimensional point cloud data of the corresponding area; as well as A processing device, operably connected to the image acquisition device and the three-dimensional scanning device, for executing the method according to any one of claims 1 to 7.

10. An electronic device comprising: at least one processor; as well as At least one memory communicatively connected to the at least one processor, the at least one memory storing instructions, which, when executed individually or collectively by the at least one processor, enable the at least one processor to execute the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein: When the computer instructions are executed individually or collectively by one or more processors of a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

12. A computer program product comprising computer instructions which, when executed individually or collectively by one or more processors of a computer, cause the computer to perform the method according to any one of claims 1 to 7.

Images