Method, device, and system for collecting iron scrap images, electronic device, and storage medium

Abstract

To provide an iron scrap image collection method, an iron scrap image collection system, an iron scrap image collection device, an electronic device, a storage medium, and a computer program.SOLUTION: An iron scrap image collection method includes the steps of: identifying one sub-region in an iron scrap down region where an iron scrap is taken down by an iron griping device and a photographing position corresponding to the sub-region, the photographing position being a position in the sub-region where the iron gripping device in the sub-region takes down the iron scrap; acquiring three-dimensional point group data of the sub-region in response to the confirmation that the iron griping device departed from the down region; determining the magnification when the sub-region is photographed on the basis of the three-dimensional point group data; and starting image collection of iron scraps of a lot on the basis of the photographing position and the magnification. An iron scrap image collection system includes an image collection device for collecting down region images, a three-dimensional scanning device which obtains three-dimensional point group data of the region by scanning the down region, and a processing device which executes the iron scrap image collection method.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the field of machine vision, and in particular to a method, an apparatus, a ferrous scrap image acquisition system, an electronic device, a computer readable storage medium and a computer program product for acquiring images of ferrous scrap. [Background technology]

[0002] With the demand for green economy, many steel mills at home and abroad use scrap iron as an important raw material for steelmaking, in order to promote the early realization of the goals of carbon peaking and carbon neutrality and to reduce the dependency on iron ore. Steel mills need to purchase a large amount of scrap iron every year, and the scrap iron comes in a huge number of types, complex material types, various shapes and sizes, and many of them contain impurities such as waste soil and oil stains. In traditional steel mills, manual quality inspection is often used to grade the purchased scrap iron, and after the scrap iron is collected, stored, and transshipped, the grading results are manually entered and uploaded into the ERP or MES system. Summary of the Invention

[0003] The present disclosure provides a method, an apparatus, a ferrous scrap image collection system, an electronic device, a computer readable storage medium and a computer program product for collecting images of ferrous scrap.

[0004] According to one aspect of the present disclosure, a method for collecting images of iron scrap is provided, the method including: identifying a sub-area in an iron scrap unloading area where the iron scrap of the lot is unloaded by an iron gripping device during a process in which the iron gripping device transfers one lot of iron scrap to an iron scrap unloading area each time, and a shooting position corresponding to the sub-area, where the shooting position represents a position in the sub-area where the iron gripping device unloads the iron scrap of the lot; acquiring three-dimensional point cloud data for the sub-area in response to determining that the iron gripping device has left the iron scrap unloading area; determining a magnification ratio for photographing the sub-area based on the three-dimensional point cloud data; and starting to collect images of the iron scrap of the lot based on the shooting position and the magnification ratio.

[0005] According to another aspect of the present disclosure, there is provided an apparatus for collecting images of iron scrap, the apparatus including: a first module for identifying a sub-area in a scrap iron unloading area where iron scrap of the lot is unloaded by an iron gripping device in a process of the iron gripping device transferring one lot of iron scrap to the scrap iron unloading area each time, and a shooting position corresponding to the sub-area, the shooting position representing a position in the sub-area where the iron gripping device unloads the iron scrap of the lot; a second module for acquiring three-dimensional point cloud data for the sub-area in response to determining that the iron gripping device has left the scrap iron unloading area; a third module for determining a magnification ratio for photographing the sub-area based on the three-dimensional point cloud data; and a fourth module for starting collecting images of the iron scrap of the lot based on the shooting position and the magnification ratio.

[0006] According to another aspect of the present disclosure, there is provided a system for collecting images of ferrous scrap, the system including an image collection device for collecting images including at least a portion of a ferrous scrap unloading area, a three-dimensional scanning device for scanning at least a portion of the ferrous scrap unloading area to obtain three-dimensional point cloud data of a corresponding area, and a processing device operably connected to the image collection device and the three-dimensional scanning device for performing the method for collecting images of ferrous scrap of the present disclosure.

[0007] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method for collecting images of ferrous scrap of the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the method for collecting images of ferrous scrap of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a computer program product including a computer program that, when executed by a processor, implements the method for collecting images of ferrous scrap of the present disclosure.

[0010] According to one or more embodiments of the present disclosure, the unloading position of the iron gripping device can be accurately positioned to ensure that each lot of iron scrap being unloaded is comprehensively photographed and the overlap of the photographed areas can be avoided, and the magnification ratio can be automatically adjusted for the corresponding photographed area to ensure that the sizes of each lot of iron scrap photographed are relatively consistent, thereby improving the accuracy of subsequent iron scrap grading.

[0011] It should be understood that the subject matter described in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be readily apparent from the following specification. [Brief description of the drawings]

[0012] The drawings illustrate the embodiments in examples, constitute a part of the specification, and together with the written description of the specification, serve to explain exemplary embodiments of the embodiments. The embodiments shown are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements.

[0013] [Figure 1] 1 is a schematic diagram of an example system capable of implementing the methods described herein, according to an embodiment of the present disclosure. [Diagram 2] 1 illustrates an exemplary flow chart of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Diagram 3] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Figure 4] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Diagram 5] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Figure 6] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Figure 7] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap, according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of an exemplary image collection scenario for ferrous scrap according to an embodiment of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an apparatus for collecting images of ferrous scrap according to an embodiment of the present disclosure; [Figure 10] FIG. 1 is a block diagram of a system for collecting images of iron scrap according to an embodiment of the present disclosure. [Figure 11] FIG. 1 is a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings. In order to facilitate understanding, various details of the embodiments of the present disclosure are included therein, but they should be considered as merely explanatory. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits the description of known functions and structures.

[0015] In this disclosure, unless otherwise specified, the use of terms such as "first", "second", etc. to describe various elements is not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some instances, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the contextual description.

[0016] The terms used in the description of various examples of the present disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless the number of elements is specifically limited. In addition, the term "and / or" used in the present disclosure covers any one of the listed items and all possible combinations.

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0018] FIG. 1 is a schematic diagram of an example system 100 capable of implementing the methods described herein, according to an example embodiment.

[0019] Referring to FIG. 1, the system 100 includes a client device 110, a server 120, and a network 130 that communicatively couples the client device 110 to the server 120.

[0020] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 may be an application program that requires downloading and installation before it can be run, or a mini-APP (lite app) as a lightweight application program. If the client application 112 is an application program that requires downloading and installation before it can be run, the client application 112 may be pre-installed and activated on the client device 110. If the client application 112 is a mini-APP, the client application 112 does not need to be installed, and the user 102 can directly run the client application 112 on the client device 110 by searching for the client application 112 (e.g., by the name of the client application 112) in a host application or by scanning a graphic code (e.g., a barcode, a QR code, etc.) of the client application 112. In some embodiments, the client device 110 may be any type of mobile computing device, including a mobile computer, a mobile phone, a wearable computing device (e.g., a head-mounted device such as a smart watch or smart glasses), or another type of mobile device. In some embodiments, client device 110 may alternatively be a stationary computing device, such as a desktop computer, a server computer, or other type of stationary computing device.

[0021] Server 120 is a server typically deployed by an Internet Service Provider (ISP) or Internet Content Provider (ICP). Server 120 can represent a single server, a cluster of servers, a distributed system, or a cloud server providing underlying cloud services (e.g., cloud database, cloud computing, cloud storage, cloud communications, etc.). It will be appreciated that although server 120 is shown in FIG. 1 communicating with only one client device 110, server 120 can provide background services to multiple client devices simultaneously.

[0022] Examples of network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks, such as the Internet. Network 130 may be a wired or wireless network. In some embodiments, technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), and the like may be used to process data exchanged over network 130. Additionally, all or some links may be encrypted using encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), and the like. In some embodiments, custom and / or proprietary data communication technologies may be used to replace or supplement the above data communication technologies.

[0023] For the purpose of the embodiment of the present disclosure, in the example of Fig. 1, the client application 112 may be a scrap iron image collection application program, which may provide various functions related to the scrap iron image collection process, such as dividing the scrap iron unloading area, selecting the reference shooting position, setting parameters of the image collection device and / or the 3D scanning device, etc. Accordingly, the server 120 may be a server used together with the scrap iron image collection application program. The server 120 may provide an online scrap iron image collection service to the client application 112 running on the client device 110. Or, the server 120 may provide a local scrap iron image collection service by the client application 112 running on the client device 110.

[0024] The system 100 of FIG. 1 may be configured and operated in a variety of ways to accommodate the various methods and apparatus described in this disclosure.

[0025] In related technology, the method of grading the scrap iron purchased by steel mills is often manual quality inspection, and after the scrap iron is collected, stored, and transshipped, the grading results are manually entered and uploaded into the ERP or MES system. However, manual quality inspection has the following disadvantages. 1) In the grading process of manual quality inspection of scrap iron, the grader needs to climb the cargo ship or transshipment vehicle carrying the scrap iron, which has blind spots for observation and makes it difficult to measure the entire unloading area of ​​the scrap iron by eyeballing, which poses a great safety risk. 2) The grading process relies entirely on the subjective experience of the grader, which is prone to misjudgment, which makes it easy to cause disputes with the scrap iron supplier, and lacks a means of retroactive verification. 3) Manual grading cannot realize all-weather automatic grading, which takes a long time for the work flow and affects the efficiency of transshipment of scrap iron.

[0026] With the continuous improvement of technology, some scrap ferrous grading methods based on machine vision also begin to participate in the grading field of scrap ferrous. However, there exists a large overlap in the areas photographed by these scrap ferrous grading methods based on machine vision, and the images of each scrap ferrous are large or small, which affects the grading accuracy of the scrap ferrous of the whole ship or vehicle.

[0027] Therefore, the present disclosure provides a more efficient and more versatile machine vision-based ferrous scrap image collection method and system, which can replace the traditional ferrous scrap grading method of visual quality inspection, realize continuous and all-weather smart collection of ferrous scrap images indoors and outdoors, and improve the efficiency and accuracy of ferrous scrap grading.

[0028] 2 is an exemplary flow chart of a method 200 for collecting images of steel scrap according to an embodiment of the present disclosure. The client or server shown in FIG. 1 may be utilized to implement the method 200 in FIG.

[0029] As shown in FIG. 2, an embodiment of the present disclosure provides a method 200 for collecting images of steel scrap, which includes the following steps 210 to 240.

[0030] In some embodiments, steps S210 to S240 may be performed during the process in which the iron gripping device transfers one lot of iron scrap to the iron scrap unloading area each time.

[0031] Step S210: identifying a sub-area in the iron scrap unloading area where a lot of iron scrap is unloaded by the iron gripping device and a shooting position corresponding to the sub-area, where the shooting position represents a position in the sub-area where the iron gripping device unloads the lot of iron scrap.

[0032] In examples, the ferrous gripping device may be an ferrous gripper or an electromagnetic suction cup, etc., although the present disclosure is not limited thereto.

[0033] In examples, the scrap iron unloading area may be a deck of a cargo ship or a loading platform of a transfer vehicle capable of loading scrap iron, and the like, although the present disclosure is not limited thereto.

[0034] The term "one lot of scrap iron" as used in this disclosure may refer to a collection of scrap iron that is transferred in one go from a scrap iron transport side to a scrap iron recovery side by a scrap iron gripping device (e.g., from a cargo ship loaded with scrap iron docked at a wharf to a scrap iron unloading area (e.g., a scrap iron transport truck, etc.) on the wharf, or vice versa).

[0035] The term "shooting position" as used in this disclosure may refer to a focal point in an image to be captured monitored by a finder (e.g., a hardware finder or a software finder) of an image capture device, i.e., an imaging focus point where a lens of the image capture device is focused in an image to be captured or a sharpest focus point in an image to be captured. As can be understood, the shooting position does not have to be located at the geometric center of the image to be captured, and the number of shooting positions in an image to be captured may be one or more, and the present disclosure is not limited thereto.

[0036] Step S220, in response to determining that the iron gripping device has left the iron scrap unloading area, obtain three-dimensional point cloud data for the sub-area.

[0037] As can be understood, since the volume of the iron gripping device is generally very large (e.g., occupies multiple sub-areas in the iron scrap unloading area), so as to increase the transfer efficiency and maximize the weight of each lot of iron scrap that is gripped (e.g., thereby reducing the workload of the iron gripping device going back and forth to transfer the iron scrap), acquiring three-dimensional point cloud data for the sub-area where the unloaded lot of iron scrap is located after the iron gripping device leaves the iron scrap unloading area can avoid the acquired three-dimensional point cloud data from including three-dimensional point cloud information about the iron gripping device, and further effectively reduce or eliminate noise in the three-dimensional point cloud data of the sub-areas.

[0038] The term "three-dimensional point cloud data" as used in this disclosure may refer to a set of points in the form of spatial coordinates for each sampling point on an object surface acquired by scanning the object with a three-dimensional scanning device (e.g., a laser radar, etc.). The three-dimensional point cloud data may not only include coordinate information, but may further include intensity information and / or angle information, etc., and the present disclosure is not limited thereto.

[0039] In an example, three-dimensional point cloud data for the sub-region may be acquired by a three-dimensional scanning device.

[0040] In an example, to save costs, two-dimensional image data for a sub-region (e.g., an image of the sub-region captured) may be acquired by an image acquisition device, and then three-dimensional point cloud data may be generated based on the two-dimensional image data by Neural Radiance Fields (NeRF) technology. The present disclosure does not limit the manner of acquiring three-dimensional point cloud data.

[0041] Step S230: determining an enlargement ratio for capturing an image of a sub-region based on the three-dimensional point cloud data;

[0042] In practice, each lot of ferrous scrap is typically unloaded at a scrap ferrous unloading area in a fixed sequence.

[0043] By way of example and not of limitation, initially, the scrap iron unloading area is empty and does not contain any lots of scrap iron to be transferred therein. Assuming that the scrap iron unloading area is divided into N non-overlapping sub-areas, where the N sub-areas completely cover the occupied area of ​​the scrap iron unloading area, the first N lots of scrap iron are unloaded into the corresponding N sub-areas, respectively, and the N lots of scrap iron form a first layer of scrap iron stack in the scrap iron unloading area. Thereafter, the N+1th to 2Nth lots of scrap iron are similarly unloaded into the corresponding N sub-areas, respectively, and the N+1th to 2Nth lots of scrap iron form a second layer of scrap iron stack in the scrap iron unloading area, and so on.

[0044] As a result, the acquired three-dimensional point cloud data for the sub-region may be used as feature information characterizing a lot of iron scrap corresponding to the sub-region, and information about the volume, weight, height, distance (e.g., the distance between the unloading position of the lot of iron scrap and a fixed point such as an image collection device) of the lot of iron scrap corresponding to the sub-region can be directly or indirectly derived from such three-dimensional point cloud data.

[0045] In an example, a magnification ratio for photographing a sub-area corresponding to a lot of iron scrap may be determined based on a distance between a location where the iron scrap is dropped off and a fixed point such as an image collection device included in the three-dimensional point cloud data. As can be understood, the greater the distance between the location where the iron scrap is dropped off and a fixed point such as an image collection device, the greater the magnification ratio required for photographing the corresponding sub-area. Of course, the present disclosure does not limit the relationship (e.g., functional relationship, etc.) between the distance between the location where the iron scrap is dropped off and a fixed point such as an image collection device, and the magnification ratio required for photographing the corresponding sub-area.

[0046] In an example, when the height of each point of a given layer of scrap iron stack does not vary significantly, the three-dimensional point cloud data acquired for a sub-area may be three-dimensional point cloud data for the entire scrap iron stack in which one lot of scrap iron corresponding to the sub-area is located, thereby reducing the workload of the three-dimensional scanning device and improving efficiency for each sub-area in the layer of scrap iron stack, and the three-dimensional point cloud data of the scrap iron stack of the layer may be acquired as three-dimensional point cloud data for each sub-area.

[0047] For example, continuing with the non-limiting example of the scrap iron stack described above, if it is assumed that the height of each point of a given layer of the scrap iron stack does not vary significantly, then for at most N sub-areas in the scrap iron stack of that layer, three-dimensional point cloud data of the scrap iron stack of that layer can be obtained as three-dimensional point cloud data for each of the at most N sub-areas, and from such three-dimensional point cloud data, it can be concluded that the height information of the shooting positions corresponding to each of the at most N sub-areas may be consistent.

[0048] Step S240: start collecting images of the lot of iron scrap based on the photographing position and the magnification.

[0049] In an example, initiating collection of images of a lot of iron scrap based on a shooting position and a magnification may include sending an instruction to an image collection device to focus on a corresponding sub-area where the iron scrap is located based on the shooting position included in the instruction, and capturing an image of the sub-area based on the magnification included in the instruction to obtain an image of the sub-area, i.e., an image of the iron scrap of the lot corresponding to the sub-area.

[0050] In the related art, there are many shortcomings in the grading method of iron scrap based on machine vision, such as the existence of large overlapping of the photographed areas, and the images of each iron scrap are large or small, all of which affect the accuracy of grading the iron scrap of the entire ship or the entire vehicle. The method 200 can not only ensure that the iron scrap of each lot to be unloaded is comprehensively photographed by accurately positioning the position where the iron gripping device unloads a lot of iron scrap, and avoid the overlapping of the photographed areas, but also automatically adjust the magnification of the photographing to the corresponding photographed area, and ensure that the sizes of the photographed images of each lot of iron scrap are relatively consistent, thereby improving the accuracy of the subsequent iron scrap grading.

[0051] 3 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap according to an embodiment of the present disclosure. As shown in FIG. 3, before the ferrous gripping device transfers each lot of ferrous scrap to the ferrous scrap unloading area, the method 200 further includes steps S201-S202.

[0052] Step S201: obtain an initial image including a steel scrap unloading area.

[0053] In an example, the initial image may be obtained from a data file in which previously collected images including the scrap ferrous unloading area are stored.

[0054] By way of example and not by way of limitation, if the scrap iron unloading area is the bed of a transfer vehicle (e.g., a truck) capable of loading scrap iron, such a data file may include previously collected overhead or near-overhead images of the bed of the transfer vehicle (e.g., a truck hopper) of various vehicle types.

[0055] In another example, before the iron gripping device grips the first lot of iron scrap to be transferred, an initial image including the iron scrap unloading area may be acquired by the image acquisition device.

[0056] As used in this disclosure, when referring to an image including a scrap ferrous unloading area, it means that the image includes the scrap ferrous unloading area, whereby an image of the scrap ferrous unloading area can be obtained by a cropping operation or the like.

[0057] Step S202: segmenting the iron scrap unloading area based on the initial image to obtain a number of sub-areas of the iron scrap unloading area.

[0058] In an example, the scrap ferrous unloading area may be segmented based on the initial image based on factors such as the average volume of scrap ferrous to be transferred for each lot grasped by the scrap ferrous gripping device, the operating mode of the arms of the scrap ferrous gripping device, and the aspect ratio of the scrap ferrous unloading area.

[0059] For example, but not by way of limitation, the scrap iron unloading area may be divided into a line of sub-areas (especially when the aspect ratio of the scrap iron unloading area is large) or into a mesh of sub-areas (e.g., when the length and width of the scrap iron unloading area are not significantly different). Of course, the scrap iron unloading area may include only one sub-area as a result of the division, for example, when the scrap iron to be transferred for each lot grasped by the iron gripping device occupies the entire area of ​​the scrap iron unloading area on average, or when the area of ​​the scrap iron unloading area itself is small and not suitable for division into two or more sub-areas. The present disclosure does not limit the division method of the scrap iron unloading area.

[0060] 4 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap according to an embodiment of the present disclosure. As shown in FIG. 4, step S210 of method 200, i.e., identifying a sub-area in the ferrous scrap unloading area where a lot of ferrous scrap is unloaded by the ferrous gripping device and a photographing position corresponding to the sub-area, may include steps S211 to S214.

[0061] Step S211: obtain a video stream recording the process of the iron gripping device transferring one lot of iron scrap to the iron scrap unloading area.

[0062] In an example, to facilitate subsequent analysis and processing of the video stream, the video stream may adopt a mainstream video file format.

[0063] In another example, a video stream may refer to a sequence of images collected consecutively in time, and the images in the sequence may adopt mainstream image file formats to facilitate subsequent image processing on these images. This disclosure does not limit the format of the video stream.

[0064] Step S212: determining the motion trajectory of the iron gripping device based on the video stream.

[0065] In an example, the motion trajectory of the iron gripping device may be determined by identifying position coordinates of the iron gripping device from the video stream.

[0066] By way of example and not limitation, the image acquisition device may be a gun-ball integrated imaging device. Thus, for example, a deep learning algorithm may determine (e.g., calculate) the motion trajectory of the iron gripping device from video frames or images captured using the global field of view of the gun-type camera in the gun-ball integrated imaging device.

[0067] In an example, the motion trajectory of the iron gripping device may include a transfer trajectory in which the iron gripping device grabs a lot of iron scrap from the iron scrap transporting side (e.g., the deck of a cargo ship loaded with iron scrap moored at a wharf) and transfers and unloads the lot of iron scrap to a iron scrap unloading area on the iron scrap recovery side, and a motion trajectory in which the iron gripping device leaves the iron scrap unloading area after completing unloading of the lot of iron scrap and returns to the iron scrap transporting side.

[0068] Step S213: determine a sub-area in the scrap iron unloading area where the scrap iron of the batch is unloaded by the scrap iron gripping device according to the motion trajectory of the scrap iron gripping device.

[0069] In an example, a sub-area in which a singular point (e.g., a turning point or a location representing a long dwell of the iron gripping device) identified in the motion trajectory of the iron gripping device is located may be determined as the sub-area in which the iron scrap for that lot is unloaded.

[0070] Step S214: Using deep learning technology, the position where the iron gripping device drops off the iron scrap of the lot is determined as the photographing position.

[0071] It should be noted that deep learning technology itself is known to those skilled in the art, and any type of deep learning technology can be used to determine the shooting position, and the present disclosure is not limited thereto and will not be described further.

[0072] As mentioned above, the photographing position refers to, for example, the imaging focus point where the lens of the image collecting device is focused in the image to be photographed or the sharpest focus point in the image to be photographed, so that the photographing position determined by using the deep learning technology can ensure that the photographed image covers the sub-area where the iron scrap of the lot is located and is better focused on the distribution center point of the iron scrap of the lot, and avoid the occurrence of unfavorable situations such as the iron scrap being out of focus in the photographed image or being focused on the edge part of the iron scrap of the lot (e.g., the part where the iron scrap has a small mass or volume).

[0073] 5 is an exemplary flow chart of additional steps of a method for collecting images of ferrous scrap according to an embodiment of the present disclosure. As shown in FIG. 5, step S220 of method 200, i.e., acquiring three-dimensional point cloud data for a sub-area in response to determining that the ferrous gripping device has left the ferrous scrap unloading area, may include steps S221-S222.

[0074] Step S221: determine, based on at least one of the video stream and the motion trajectory of the iron gripping device, that the iron gripping device has left the iron scrap unloading area.

[0075] In an example, based on the video stream captured in step S211, it may be determined that the iron gripping device has left the iron scrap unloading area.

[0076] In another example, it may be determined that the iron gripping device has left the iron scrap unloading area based on the motion trajectory of the iron gripping device determined in step S212.

[0077] In yet another example, it may be determined that the iron gripping device has left the iron scrap unloading area based on both the video stream and the motion trajectory of the iron gripping device.

[0078] Step S222: start scanning the sub-region using a 3D scanning device to obtain 3D point cloud data of the sub-region.

[0079] As previously discussed, scanning a sub-area utilizing a three-dimensional scanning device may include scanning only the corresponding sub-area or scanning the entire scrap ferrous stack layer including the corresponding sub-area, and the present disclosure is not limited thereto.

[0080] Thus, acquiring three-dimensional point cloud data for the sub-area where the unloaded iron scrap is located after the iron gripping device leaves the iron scrap unloading area can avoid three-dimensional point cloud information about the iron gripping device from being included in the acquired three-dimensional point cloud data, and further effectively reduce or eliminate noise in the three-dimensional point cloud data of the sub-area, and improve the accuracy of the acquired three-dimensional point cloud data.

[0081] 6 is an exemplary flowchart of additional steps of a method for collecting images of steel scrap according to an embodiment of the present disclosure. As shown in FIG. 6, step S230 of method 200, i.e., determining a magnification ratio for photographing a sub-region based on the 3D point cloud data, may include steps S231 to S234.

[0082] Step S231: Perform surface reconstruction on one lot of iron scrap in the sub-region using the three-dimensional point cloud data to obtain a surface mesh of the iron scrap of the lot.

[0083] It should be noted that the use of 3D point cloud data of an object to perform surface reconstruction on the object is known to those skilled in the art, and any type of 3D point cloud processing means can be used to determine the surface mesh of a lot of iron scrap, and the present disclosure is not limited thereto and will not be described further.

[0084] In an example, based on the resulting surface mesh of the lot of ferrous scrap, an average height of the top surface of the lot of ferrous scrap may be calculated.

[0085] Step S232: determine a height from the top surface of the iron scrap of the lot to the ground surface of the sub-region based on the surface mesh of the iron scrap of the lot and the ground point cloud data included in the 3D point cloud data.

[0086] The term "ground surface" as used in this disclosure is not limited to the earth plane and may refer to any given suitable reference plane.

[0087] By way of example and not limitation, the ground of a sub-area may refer to the bottom surface of the scrap steel unloading area, and in the case of a truck hopper, the ground may refer to the bottom inner surface or bottom outer surface of the hopper.

[0088] As can be appreciated, when a ground surface of a sub-region is given, the distance between a detection port (e.g., a lens, a radiation transmitting / receiving port, etc.) of an electronic device such as an image collection device and / or a three-dimensional scanning device and the like and the ground surface may be measurable.

[0089] As can be further appreciated, the distance between the detection port of an electronic device, such as an image collection device and / or a three-dimensional scanning device, and the ground is typically greater than the distance between the top surface of any lot of ferrous scrap and the ground (i.e., the height of the sub-area from the top surface of the lot of ferrous scrap to the ground) because the detection port of the electronic device needs to be able to perform image collection and / or radiation scanning of the sub-area from an overhead or near overhead perspective.

[0090] In an example, an average ground height of the sub-region may be calculated based on ground point cloud data included in the three-dimensional point cloud data.

[0091] Step S233: Determine a distance between the image acquisition device and the image acquisition position based on the position parameters of the image acquisition device capturing the image of the sub-area relative to the scrap iron unloading area and the height of the sub-area from the top surface of the scrap iron of the lot to the ground.

[0092] In an example, the positional parameters of the image collection device relative to the scrap iron unloading area may include the distance H between the lens of the image collection device and the ground of the sub-area (e.g., the bottom of the scrap iron unloading area) (i.e., the height from the image collection device to the ground of the sub-area) and the distance d from the shooting position to a perpendicular line of the ground passing through the image collection device.

[0093] For example, but not by way of limitation, if the height from the top surface of a given lot of ferrous scrap to the ground in the sub-area in which the lot of ferrous scrap is located is h, the distance s between the image acquisition device and the capture position can be expressed as:

number

[0094] Step S234: determining a magnification ratio for photographing the sub-region based on at least the distance between the image acquisition device and the photographing position;

[0095] In an example, the magnification at which the sub-region is imaged may be determined by varying the distance between the image acquisition device and the image capture location.

[0096] As mentioned above, the greater the distance between the unloading position of the iron scrap and a fixed point such as the image collecting device, the greater the magnification required for photographing the corresponding sub-area. Thus, in order to select an appropriate magnification for photographing the corresponding sub-area, a function formula may be constructed such that, for example, the magnification of the photograph increases as the distance between the image collecting device and the photographing position increases.

[0097] 7 is an exemplary flow chart of additional steps of a method for collecting images of steel scrap according to an embodiment of the present disclosure. As shown in FIG. 7, step S234, i.e., determining a magnification ratio for capturing images of the sub-area based on at least a distance between the image collection device and the capturing position, may include steps S2341 to S2343.

[0098] Step S2341: determining a reference photographing position in the iron scrap unloading area and a reference distance between the image acquisition device and the reference photographing position.

[0099] In an example, the reference photography location may be a point in the scrap ferrous unloading area (e.g., the hopper of a transfer truck) where no transfer lots of scrap ferrous are loaded. In other words, the reference photography location may be established prior to transfer of each lot of scrap ferrous.

[0100] For example, but not by way of limitation, when the bottom inner surface of the hopper is the ground, the reference shooting position may be a point on the bottom inner surface of the hopper. If the distance between the lens of the collecting device and the ground is H0, and the distance from the reference shooting position to the perpendicular line of the ground passing through the image collecting device is d0, the reference distance s0 between the image collecting device and the reference shooting position is s0=sqrt(H0 2 +d0 2 ) can be determined.

[0101] As can be appreciated, the reference shooting positions may be selected by one of ordinary skill in the art as needed, and the present disclosure is not limited thereto.

[0102] Step S2342: focus the image collection device on the reference shooting position, and adjust the magnification of the image collection device so that the image collection device can capture just the entire area of ​​the scrap iron unloading area, and obtain the reference magnification of the image collection device.

[0103] As can be appreciated, the fact that the image collection device can capture just the entire area of ​​the scrap ferrous unloading area may in some cases mean that the boundary of the image captured by the image collection device just approximately overlaps with the periphery of the scrap ferrous unloading area (e.g., the frame of the truck hopper), thereby obtaining a reference magnification factor k0.

[0104] Step S2343: determine the magnification ratio for photographing the sub-region based on the distance between the image acquisition device and the photographing position, the reference distance and the reference magnification ratio.

[0105] By way of example, and not by way of limitation, the magnification factor k for capturing an image of a sub-region is given by:

number

[0106] As can be understood, the above describes the determination of the magnification factor k for photographing a sub-region in the form of a linear function, but a person skilled in the art may select any suitable function formula as needed to determine the magnification factor of the image acquisition device for photographing different sub-regions, and the present disclosure is not limited thereto.

[0107] In some embodiments, initiating the collection of images of the lot of ferrous scrap based on the imaging position and the magnification may include focusing an image collection device on the imaging position, adjusting a magnification of the image collection device to the magnification, and capturing an image of a sub-area to obtain an image of the lot of ferrous scrap.

[0108] This allows the magnification of the photography to be automatically adjusted for the corresponding photography area, ensuring that the sizes of the photographed images of each lot of iron scrap are relatively consistent, thereby improving the accuracy of subsequent iron scrap grading.

[0109] FIG. 8 is a schematic diagram of an exemplary image collection scenario for steel scrap according to an embodiment of the present disclosure.

[0110] It should be noted that although FIG. 8 illustrates an image collection scenario of scrap iron during unloading by a harbor crane at a quay, one skilled in the art will appreciate that the image collection method of scrap iron disclosed herein may be applied to any other suitable scrap iron transshipment scenario, and the disclosure is not limited thereto.

[0111] The image collection scenario for iron scrap shown in Figure 8 includes an iron scrap transshipment equipment 1, a pole bracket equipment 2, an image collection device 3, a terminal card reader 4, an LED display device 5, an image processing device 6, a cabin 7, a transshipment platform 8, a gunball-integrated camera 9, a supplementary light 10, a port crane base 11, and an iron grabber 12.

[0112] As shown in the figure, the scrap iron transfer device 1 may be deployed on the quayside and is responsible for grabbing (or sucking) the scrap iron in the cabin 7 and transferring it into the transfer platform 8. The pole bracket device 2 may be deployed near the parking space so that the camera faces the scrap iron transfer vehicle as directly as possible. The image collection device 3 (including the gunball-integrated camera 9 and the supplementary light 10, etc.) and the three-dimensional scanning device 13 may be mounted on the top of the pole bracket device 2. Note that the terminal card reader 4 may be deployed at the bottom of the pole bracket device 2, and the LED display device 5 may be installed in the middle of the pole bracket device 2. The image processing device 6 may be remotely deployed in a server room and connected to the image collection device 3, the terminal card reader 4, the LED display device 5 and / or the three-dimensional scanning device 13 via a communication circuit (e.g., a wired communication circuit or a wireless communication channel or a combination thereof).

[0113] As shown in the figure, the pole bracket device 2 may have an inverted "L" shape, and on the top beam there are a mounting bracket for a gunball-integrated camera 9 in the image collection device 3, a mounting bracket for a supplementary light 10, etc. The pole bracket device 2 may adopt a hollow design, and all communication lines and power supply lines are wired and connected inside the bracket, thereby avoiding the lines from being exposed to a complex environment and improving safety and durability.

[0114] As further shown in the figure, the scrap iron transfer device 1 may be composed of a port crane platform 11, a scrap iron grabber 12, etc. The operator controls the interval at which the scrap iron grabber 12 transfers each lot of scrap iron according to the command displayed on the LED display device 5, and sequentially grabs the scrap iron in the cabin 7 and lowers it into the transfer platform 8.

[0115] As can be seen, the gun-type camera in the gun-ball integrated camera 9 can be responsible for controlling the global view of the unloading area by the port crane at the wharf, and the ball-type camera can be responsible for taking and collecting the scrap iron pictures in the transfer platform 8. The supplementary light 10 can meet the lighting demand during nighttime work, and can ensure the imaging quality of the scrap iron pictures even during the day by reducing the impact of the shadow caused by the oblique sunlight on the transfer platform 8 on the imaging of the gun-ball integrated camera 9.

[0116] As further shown in the figure, a terminal card reader 4 may be installed at the bottom of the pole bracket device 2 and communicatively connected to other assemblies via communication circuitry within the pole bracket, thereby allowing an operator (e.g., a transfer vehicle driver) to initiate the smart collection and automatic grading process of scrap iron images.

[0117] Optionally, the LED display device 5 may be installed in the center of the pole bracket device 2, which is communicatively connected to other assemblies via a communication circuit inside the pole bracket, and can display text information in various colors (e.g., displaying hints for operations that the operator of the scrap iron transfer equipment 1 needs to perform), and different information may be displayed in different colors according to type.

[0118] The image processing device 6 may include an image processing server as the core processing unit of the smart collection and automatic grading process of scrap metal images, which performs information interaction and control with each assembly at the wharf site via a communication line.

[0119] In an example, the image processing device 6 can automatically start related task flows by receiving the logistics code information, the smart collection of scrap iron images and the start command of the automatic grading process transmitted by the terminal card reader 4. After all task flows are completed, the overall grading result of the transshipment scrap iron on the vehicle may be displayed on the interface of the terminal card reader 4 for the operator to view.

[0120] Optionally, interaction information such as logistics code information transmitted by the terminal card reader 4, smart collection of scrap iron images and instructions to start the automatic grading process may alternatively be realized using a handheld terminal and associated applications.

[0121] Optionally, if interaction information, such as logistics code information that needs to be transmitted, smart collection of scrap iron images and starting commands for the automatic grading process, can be obtained directly from the contractor's traditional MES or ERP system interface, there is no need to install the terminal card reader 4.

[0122] Optionally, before each lot of iron scrap is transferred to the transfer platform 8, an image identification algorithm can be used to determine whether the parking position of the platform 8 of the transfer vehicle is within the best shooting range of the image collection device 3, and if not, guide the parking of the platform 8. For example, if the parking position of the vehicle does not meet the requirements, a presentation information can be generated and automatically fed back to the on-site LED display device 5 to present to the operator (e.g., the driver) that the parking position of the vehicle does not meet the requirements and needs to be re-adjusted for parking, until it is determined that the vehicle is parked near the best position.

[0123] Optionally, a photograph of each collected lot of ferrous scrap may be transmitted to the image processing device 6 and stored.

[0124] Accordingly, the computer vision algorithm in the image processing device 6 can perform discrimination on the collected images of the iron scrap from multiple feature dimensions such as thickness, material type, size, diameter, density fitting, non-steel impurities and sealant hazards, etc., to obtain a comprehensive evaluation level of the iron scrap to be transshipped in the entire vehicle. Additionally, when the image processing device 6 detects the existence of sealant hazards in the images of the iron scrap, the alert text information can be transmitted to the LED display device 5 via the communication circuit, so as to promptly alert the operator of the iron scrap transshipment device 1 to take corresponding measures.

[0125] FIG. 9 is a block diagram of an apparatus for collecting images of steel scrap according to an embodiment of the present disclosure.

[0126] As shown in FIG. 9 , according to an embodiment of the present disclosure, an apparatus 900 for collecting images of iron scrap is provided, the apparatus 900 including: a first module 910 for identifying a sub-area in a scrap iron unloading area where a lot of iron scrap is unloaded by an iron gripping device and a photographing position corresponding to the sub-area, where the photographing position represents a position in the sub-area where the iron gripping device unloads the lot of iron scrap; a second module 920 for acquiring three-dimensional point cloud data for the sub-area in response to determining that the iron gripping device has left the scrap iron unloading area; a third module 930 for determining a magnification for photographing the sub-area based on the three-dimensional point cloud data; and a fourth module 940 for starting collecting images of the lot of iron scrap based on the photographing position and the magnification.

[0127] Here, the operations of the above-mentioned modules 910-940 of the apparatus 900 are similar to those of steps 210-240 of the above-mentioned method 200, respectively, and will not be described here. In some embodiments, the apparatus 900 may further include one or more additional modules or units for performing functions corresponding to additional steps of the method 200 (e.g., steps 201-202, 211-214, 221-222, 231-234, steps 2341-2343, etc.), and for the sake of simplicity of illustration, these modules or units are not shown in FIG.

[0128] In the related art, there are many shortcomings in the grading method of iron scrap based on machine vision, such as the existence of large overlapping of the photographed areas, and the images of each iron scrap are large or small, which all affect the accuracy of the grading of the iron scrap of the entire ship or the entire vehicle. The device 900 can not only ensure that the iron scrap of each lot to be unloaded is comprehensively photographed by accurately positioning the position where the iron gripping device unloads a lot of iron scrap, and avoid the overlapping of the photographed areas, but also automatically adjust the magnification of the photographing to the corresponding photographed area, and ensure that the size of the photographed images of each lot of iron scrap is relatively consistent, thereby improving the accuracy of the subsequent iron scrap grading.

[0129] In some embodiments, the apparatus 900 may further include a fifth module for acquiring an initial image including the scrap iron unloading area before the iron gripping device transfers each lot of scrap iron to the scrap iron unloading area, and for segmenting the scrap iron unloading area based on the initial image to acquire multiple sub-areas of the scrap iron unloading area.

[0130] In some embodiments, the first module 910 of the apparatus 900 may include: a first unit for acquiring a video stream recording a process of the iron gripping device transferring a lot of iron scrap to the iron scrap unloading area; a second unit for determining a movement trajectory of the iron gripping device based on the video stream; a third unit for determining a sub-area in the iron scrap unloading area where the lot of iron scrap is unloaded by the iron gripping device based on the movement trajectory of the iron gripping device; and a fourth unit for using deep learning technology to determine the position where the iron gripping device unloads the lot of iron scrap as the shooting position.

[0131] In some embodiments, the second module 920 of the apparatus 900 may include a fifth unit for determining that the iron gripping device has left the iron scrap unloading area based on at least one of the video stream and the motion trajectory of the iron gripping device, and a sixth unit for starting to scan the sub-area using the three-dimensional scanning device to obtain three-dimensional point cloud data of the sub-area.

[0132] In some embodiments, the third module 930 of the apparatus 900 may include a seventh unit for performing surface reconstruction on a lot of iron scrap in the sub-area using the three-dimensional point cloud data to obtain a surface mesh of the iron scrap of the lot; an eighth unit for determining a height from a top surface of the iron scrap of the lot to a ground surface of the sub-area based on the surface mesh of the iron scrap of the lot and the ground point cloud data included in the three-dimensional point cloud data; a ninth unit for determining a distance between the image collection device and an imaging position based on position parameters of an image collection device for imaging the sub-area relative to the iron scrap unloading area and the height from the top surface of the iron scrap of the lot to the ground surface of the sub-area; and a tenth unit for determining a magnification ratio for imaging the sub-area based on at least the distance between the image collection device and the imaging position.

[0133] In some embodiments, the tenth unit may be further configured to determine a reference shooting position in the scrap iron unloading area and a reference distance between the image collection device and the reference shooting position, focus the image collection device on the reference shooting position, adjust the magnification of the image collection device so that the image collection device just captures the entire area of ​​the scrap iron unloading area, obtain a reference magnification of the image collection device, and determine a magnification for photographing the sub-area based on the distance between the image collection device and the shooting position, the reference distance, and the reference magnification.

[0134] In some embodiments, the fourth module 940 of the apparatus 900 may be further configured to focus the image capture device on the capture position, adjust the magnification of the image capture device to the magnification, and capture a sub-area to obtain an image of the scrap iron of the lot.

[0135] Although particular functions are discussed above with reference to particular modules, it should be noted that the functionality of each module discussed herein may be split into multiple modules and / or at least some of the functionality of multiple modules may be combined into a single module. A particular module performing an operation, as discussed herein, may include the particular module itself performing the operation or, alternatively, the particular module calling or otherwise accessing another assembly or module that performs the operation (or performs the operation in conjunction with the particular module). Thus, a particular module performing an operation may include the particular module itself performing the operation and / or another module performing an operation that the particular module calls or otherwise accesses.

[0136] It should be further understood that the present specification may describe various techniques in the general context of software or hardware elements or program modules. Each module described in FIG. 9 above may be implemented in hardware or hardware combining software and / or firmware. For example, the modules may be implemented as computer program code / instructions configured to be executed on one or more processors and stored on a computer-readable storage medium. Alternatively, the modules may be implemented as hardware logic / circuitry. The hardware logic / circuitry may include integrated circuit chips (e.g., a processor (e.g., a Central Processing Unit (CPU), a microcontroller, a microprocessor, a Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0137] 10 is a block diagram of an image collection system for iron scrap according to an embodiment of the present disclosure. As shown in FIG. 10, the image collection system for iron scrap 1000 may include an image collection device 1010 for collecting an image including at least a portion of an iron scrap unloading area, a three-dimensional scanning device 1020 for scanning at least a portion of the iron scrap unloading area to obtain three-dimensional point cloud data of the corresponding area, and a processing device 1030 operably connected to the image collection device 1010 and the three-dimensional scanning device 1020 to perform a method for collecting images of iron scrap of the present disclosure, such as the method 200 and additional steps of each embodiment thereof.

[0138] As can be appreciated, the image acquisition device 1010 may include any suitable type of camera, video camera, or the like, and the present disclosure is not limited thereto. The three-dimensional scanning device 1020 may include any suitable type of commercially available three-dimensional scanning equipment, such as a radar or laser radar, and the present disclosure is not limited thereto. The processing device 1030 may be any customized or commercially available processor, such as a central processing unit (CPU), a distributed processing unit, a semiconductor-based (e.g., in the form of a microchip or chipset) microprocessor, and the like, and the present disclosure is not limited thereto.

[0139] According to embodiments of the present disclosure, an electronic device, a readable storage medium, and a computer program product are further provided.

[0140] According to an embodiment of the present disclosure, there is provided an electronic device, which may include at least one processor and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method for collecting images of ferrous scrap of the present disclosure.

[0141] According to an embodiment of the present disclosure, there is provided a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions may be used to cause a computer to perform a method for collecting images of ferrous scrap of the present disclosure.

[0142] According to an embodiment of the present disclosure, there is provided a computer program product, which may include a computer program that, when executed by a processor, is capable of implementing the method for collecting images of ferrous scrap of the present disclosure.

[0143] As can be seen, the scrap iron image collection method disclosed herein has the advantages of being versatile, automatic, and does not require human intervention. The scrap iron image collection method disclosed herein can accurately locate the unloading position, ensure that more scrap iron is comprehensively photographed, and effectively reduce the probability of photographing the same location multiple times, thereby significantly improving the photographing coverage, reducing the overlapping rate of the photographed area, and making the grading of the scrap iron in the entire vehicle more accurate. In addition, the scrap iron image collection method disclosed herein can self-adaptively adjust the magnification of the photographing, ensuring the stability of the image size of each scrap iron photographed, and improving the accuracy of identification.

[0144] Referring to FIG. 11, a block diagram of an electronic device 1100 that can be used as a server or client of the present disclosure is described, which is an example of a hardware device that can be applied to each aspect of the present disclosure. The electronic device represents various types of digital electronic computing devices, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital processing, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] 11, the electronic device 1100 includes a computing unit 1101, which can perform various suitable operations and processes based on a computer program stored in a read-only memory (ROM) 1102 or loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may further store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, the ROM 1102 and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0146] A number of components in the electronic device 1100 are connected to the I / O interface 1105, and include an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the electronic device 1100, and the input unit 1106 may receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 1107 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 1109 enables the electronic device 1100 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0147] The computing unit 1101 may be any of a variety of general-purpose and / or special-purpose processing assemblies having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning network algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 executes the methods and processes described above, such as method 200 and its embodiments. For example, in some embodiments, the method 200 and its embodiments may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the method 200 described above and additional steps of its embodiments may be executed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform additional steps of the method 200 and its respective embodiments in any other suitable manner (eg, by firmware).

[0148] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device;

[0149] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes may be entirely executed on a machine, partly executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0150] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in combination with an instruction execution system, device or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more wire-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0151] To provide for interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse and trackball) by which the user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input may be received from the user in any form (including sound input, speech input, or tactile input).

[0152] The systems and techniques described herein may be implemented in a computing system including a backstage component (e.g., a data server, etc.), a middleware component (e.g., an application server, etc.), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with the system or technique embodiment), or any combination of the backstage component, middleware component, or front-end component. The components of the system may be interconnected (e.g., via a communications network) by any form or medium of digital data communication. Examples of communications networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0153] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other via a communication network. The client and the server are created by running computer programs having a client-server relationship with each other on corresponding computers. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0154] It should be understood that steps may be reordered, added, or removed using the various types of flows shown above. For example, each step described in this disclosure may be performed in parallel, sequentially, or in a different order, and is not limited herein, as long as the technical solution disclosed in this disclosure can achieve the desired results.

[0155] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents after grant. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In essence, with the evolution of technology, many elements described herein may be replaced by equivalent elements appearing later in this disclosure.

Claims

1. 1. A method for collecting images of ferrous scrap, performed by an apparatus for collecting images of ferrous scrap, comprising: During the process of the iron grabbing device transferring one lot of iron scrap to the iron scrap unloading area each time, the first module: identifying a sub-area within the scrap ferrous unloading area where the scrap ferrous of the lot is unloaded by the scrap ferrous gripping device and a photographing location corresponding to the sub-area, the photographing location representing a location within the sub-area where the scrap ferrous of the lot is unloaded by the scrap ferrous gripping device; acquiring, by a second module, in response to determining that the ferrous gripping device has left the ferrous scrap unloading area, three-dimensional point cloud data for the sub-area; determining, by a third module, a magnification ratio for photographing the sub-region based on the three-dimensional point cloud data; and starting, by a fourth module, collecting images of the lot of iron scrap based on the photographing position and the magnification. method.

2. Before the iron gripping device transfers each lot of iron scrap to the iron scrap unloading area, the method includes: acquiring an initial image including the scrap iron unloading area; The method of claim 1 , further comprising: segmenting the scrap ferrous drop-off area based on the initial image to obtain a plurality of sub-areas of the scrap ferrous drop-off area.

3. The step of identifying a sub-area in the scrap iron unloading area where the scrap iron of the lot is unloaded by the scrap iron gripping device and a photographing position corresponding to the sub-area includes: acquiring a video stream recording the process of the iron gripping device transferring the lot of iron scrap to the iron scrap unloading area; determining a motion trajectory of the iron gripping device based on the video stream; determining the sub-area within the iron scrap unloading area where the iron scrap of the lot is unloaded by the iron gripping device based on the motion trajectory of the iron gripping device; and determining, using deep learning techniques, as the photographing position, a position where the iron gripping device unloads the iron scrap of the lot.

4. In response to determining that the iron gripping device has left the iron scrap unloading area, acquiring three-dimensional point cloud data for the sub-area includes: determining that the iron gripping device has left the iron scrap unloading area based on at least one of the video stream and the motion trajectory of the iron gripping device; and commencing scanning the sub-region using a three-dimensional scanning device to obtain three-dimensional point cloud data of the sub-region.

5. The step of determining a magnification ratio for photographing the sub-region based on the three-dimensional point cloud data includes: performing surface reconstruction on the iron scrap of the lot in the sub-region using the 3D point cloud data to obtain a surface mesh of the iron scrap of the lot; determining a height from a top surface of the iron scrap of the lot to a ground surface of the sub-area based on the surface mesh of the iron scrap of the lot and ground point cloud data included in the three-dimensional point cloud data; determining a distance between the image acquisition device and the image capture position based on a position parameter of an image acquisition device capturing the image of the sub-area relative to the ferrous scrap unloading area and the height of the sub-area from the top surface of the ferrous scrap of the lot to the ground; determining the magnification at which to image the sub-region based on at least the distance between the image acquisition device and the image capture position.

6. determining the magnification for capturing the image of the sub-region based on at least the distance between the image acquisition device and the image capturing position, determining a reference photographing position in the scrap iron unloading area and a reference distance between the image acquisition device and the reference photographing position; Making the image acquisition device focus on the reference photographing position and adjusting the magnification of the image acquisition device so that the image acquisition device can capture just the entire area of ​​the iron scrap unloading area, and obtaining a reference magnification of the image acquisition device; determining the magnification for photographing the sub-region based on the distance between the image acquisition device and the photographing position, the reference distance and the reference magnification.

7. The step of starting to collect images of the lot of iron scrap based on the photographing position and the magnification factor includes:

2. The method of claim 1, comprising the steps of focusing an image capture device at the capture location, adjusting a magnification of the image capture device to the magnification, and capturing an image of the sub-area to obtain an image of the lot of scrap iron.

8. 1. An apparatus for collecting images of ferrous scrap, comprising: a first module for identifying a sub-area in the iron scrap unloading area where the iron scrap of the lot is unloaded by the iron gripping device and a photographing position corresponding to the sub-area during a process in which the iron gripping device transfers one lot of iron scrap to the iron scrap unloading area each time, the photographing position representing a position in the sub-area where the iron gripping device unloads the iron scrap of the lot; a second module for acquiring three-dimensional point cloud data for the sub-area in response to determining that the ferrous gripping device has left the ferrous scrap unloading area; a third module for determining a magnification ratio for capturing the sub-region based on the three-dimensional point cloud data; and a fourth module for initiating collecting images of the lot of iron scrap based on the photographing position and the magnification. Device.

9. 1. A system for collecting images of steel scrap, comprising: an image capture device for capturing an image including at least a portion of the scrap ferrous unloading area; a three-dimensional scanning device for scanning at least a portion of a steel scrap unloading area to obtain three-dimensional point cloud data of a corresponding area; A system for collecting images of iron scrap, comprising: a processing device operably connected to said image collection device and said three-dimensional scanning device for performing the method of any one of claims 1 to 7.

10. An electronic device, At least one processor; and at least one memory communicatively connected to said at least one processor, said at least one memory storing instructions which, when executed alone or in concert by said at least one processor, cause said at least one processor to perform the method of any one of claims 1 to 7. electronic equipment.

11. A non-transitory computer readable storage medium having stored thereon computer instructions which, when executed alone or in concert by one or more processors of a computer, cause the computer to perform the method of any one of claims 1 to 7.

12. A computer program comprising computer instructions which, when executed alone or in concert by one or more processors of a computer, cause the computer to carry out a method according to any one of claims 1 to 7.

Images