Detection method, detection device, security inspection equipment, electronic equipment, medium and program product

Abstract

The detection method comprises the steps that when a person is located in an entrance area of a channel, a first area of the person is shot to obtain a first image; after the person enters the channel through the entrance area, shooting a second area of the person to obtain a second image; performing multi-view fusion on the first image and the second image to obtain a third image; the first image, the second image and the third image are detected, to-be-detected objects carried by the personnel are recognized, and the to-be-detected objects comprise objects to be subjected to radiation scanning by detection equipment beside the channel.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, radiation detection or other technical fields, and more specifically, to a detection method, a detection device, security inspection equipment, electronic equipment, a medium and a program product. Background Art

[0002] In some public places (such as airports, stations or customs, etc.), there are clear requirements for security checks. Taking customs as an example, customs has clear regulatory requirements for international inbound passengers' biosafety, carry-on luggage safety, nuclear radiation safety and other aspects. Passengers need to go through multiple monitoring and inspection links such as health declaration, temperature detection, health declaration verification, nuclear radiation detection, carry-on luggage inspection, risk analysis, etc.

[0003] For the carry-on baggage inspection, passengers need to place their baggage on the radiation imaging device, and then wait for the radiation imaging device to transport the baggage to the radiation scanning area, and then to the baggage collection area. After the radiation imaging device outputs the image, it still needs to be manually interpreted, which generally takes a few seconds to more than ten seconds depending on the complexity of the image. It can be seen that it takes some time for a baggage to be checked after the passenger places it. During this time, the passenger needs to wait.

[0004] In the process of implementing the invention concept of this application, the inventor found that passengers without luggage do not need to undergo carry-on luggage inspection, so it is necessary to manually determine whether each passenger is carrying luggage. This method consumes manpower, and when the passenger flow is large, the number of manual workers and the speed of judgment are high, which is likely to affect the efficiency of equipment use, security inspection efficiency and passenger pass rate, and may reduce user experience. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a detection method, a detection device, a security inspection device, an electronic device, a medium and a program product.

[0006] According to a first aspect of the present disclosure, a detection method is provided, comprising: when a person is located in an entrance area of ​​a passage, photographing a first area of ​​the person to obtain a first image; after the person passes through the entrance area and enters the passage, photographing a second area of ​​the person to obtain a second image; performing multi-perspective fusion on the first image and the second image to obtain a third image; and detecting the first image, the second image, and the third image respectively to identify an object to be inspected carried by the person, wherein the object to be inspected includes an object to be subjected to radiation scanning by a detection device next to the passage.

[0007] In some embodiments, performing multi-perspective fusion of the first image and the second image to obtain a third image includes: splicing and fusing images of persons displayed at different perspectives in the first image and the second image to obtain the third image.

[0008] In some embodiments, performing multi-perspective fusing of the first image and the second image to obtain a third image further includes: obtaining N first images taken at N moments, and N second images taken at N moments, where N is an integer greater than or equal to 2; performing multi-perspective fusing of the N first images and the N second images one by one to obtain N third images.

[0009] In some embodiments, the N moments form a time sequence with a chronological order.

[0010] In some embodiments, for the first image and the second image fused at each moment, each image of the person is displayed from a different perspective, and the perspectives are complementary.

[0011] In some embodiments, photographing a first area of ​​the person to obtain a first image includes: photographing a first area of ​​the person to obtain a first video stream; wherein the N first images include N image frames at the N moments extracted from the first video stream.

[0012] In some embodiments, photographing the second area of ​​the person to obtain a second image includes: photographing the second area of ​​the person to obtain a second video stream; wherein the N second images include N image frames at the N moments extracted from the second video stream.

[0013] In some embodiments, the first image, the second image and the third image are detected respectively, and identifying the object to be inspected carried by the person includes: detecting each of the N first images, the N second images and the N third images, and identifying the object to be inspected carried by the person.

[0014] In some embodiments, identifying the object to be inspected carried by the person includes: in a target image having the person and the object to be inspected, selecting the identified person and the object to be inspected to obtain a person frame and at least one object frame, wherein the target image includes at least one of the first image, the second image and the third image; and determining at least one object to be inspected carried by the person based on the spatial relationship between the person frame and each of the object frames.

[0015] In some embodiments, the identity of the person is bound to the object identity of each object to be inspected that the person carries.

[0016] In some embodiments, when there are multiple target images, identifying the object to be inspected carried by the person includes: cropping each image of the object to be inspected in each target image to obtain an image set; and marking images in the image set belonging to the same object to be inspected as the same object identifier.

[0017] In some embodiments, marking images in the image set belonging to the same object to be inspected as the same object identification includes: extracting image features for each image of the object to be inspected in the image set to obtain a feature set; for any image feature in the feature set, calculating the similarity between it and the remaining image features, and if the similarity is greater than or equal to a preset value, marking the images corresponding to the two image features as the same object identification.

[0018] In some embodiments, for any image feature in the feature set, if the similarity between it and the remaining image features in the current feature set is less than the preset value, a new object identifier is generated to mark the corresponding image.

[0019] In some embodiments, an entrance access device is provided at the entrance area, and M1 first cameras are installed on the entrance access device for photographing the first area of ​​the personnel, where M1 is an integer greater than or equal to 1.

[0020] In some embodiments, M2 second cameras are installed on the entrance access device for photographing the second area of ​​the person, where M2 is an integer greater than or equal to 1.

[0021] In some embodiments, after identifying the object to be inspected carried by the person, the method further includes: when the person is located in the exit area of ​​the passage, photographing the first area to obtain a fourth image; detecting the fourth image to identify the object to be inspected that has not been scanned by radiation from the detection device.

[0022] In some embodiments, after identifying the object to be inspected carried by the person, the method further includes: photographing a second area of ​​the person in the channel to obtain a fifth image, and identifying the object to be inspected that has not been radiated and scanned by the detection device.

[0023] In some embodiments, an exit passage device is provided in the exit area, and M3 third cameras are installed on the exit passage device for photographing the first area of ​​the personnel, where M3 is an integer greater than or equal to 1.

[0024] In some embodiments, the detection device includes a placement area, a scanning area and a picking area, wherein the object to be inspected after radiation scanning is configured to be transported from the placement area to the scanning area for radiation scanning, and then transported to the picking area; wherein the placement area is located inside the channel, and the picking area is located outside the channel.

[0025] Another aspect of an embodiment of the present disclosure provides a detection device, including: a first shooting module, used to shoot a first area of ​​the person to obtain a first image when the person is located in the entrance area of ​​the passage; a second shooting module, used to shoot a second area of ​​the person to obtain a second image after the person passes through the entrance area and enters the passage; an image fusion module, used to perform multi-perspective fusion of the first image and the second image to obtain a third image; an object detection module, used to detect the first image, the second image and the third image respectively, and identify the object to be inspected carried by the person, wherein the object to be inspected includes an object to be radiation scanned by the detection equipment next to the passage.

[0026] Another aspect of an embodiment of the present disclosure provides a security inspection device, including: an entrance pass device; an exit pass device, defining a channel between the exit pass device and the entrance pass device; a detection device, located on one side of the channel; and an electronic device, used to implement the detection method as described in any one of the above items.

[0027] In some embodiments, the security inspection equipment further includes M1 first cameras installed at the entrance access device, and used to be controlled by the electronic device to photograph a first area of ​​a person, where M1 is an integer greater than or equal to 1.

[0028] In some embodiments, the security inspection equipment also includes M2 second cameras installed at the entrance access device, which are used to be controlled by the electronic device to photograph a second area of ​​the personnel, and M2 is an integer greater than or equal to 1.

[0029] In some embodiments, the security inspection equipment also includes M3 third cameras installed in the exit passage device, which are used to be controlled by the electronic device to photograph the first area of ​​the personnel, and M3 is an integer greater than or equal to 1.

[0030] Another aspect of an embodiment of the present disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0031] Another aspect of an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor executes the method as described above.

[0032] Another aspect of an embodiment of the present disclosure provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0033] One or more of the above embodiments have the following beneficial effects: by using images taken from two different perspectives in the entrance area and the passage, as well as images fused from multiple perspectives, it is possible to quickly identify whether a passerby is carrying an object to be inspected, which not only reduces the reliance on manual judgment, reduces the risk of misjudgment and missed inspection, but also improves the accuracy and safety of security inspection. In addition, passersby with luggage are identified to prevent missed inspection, and passersby without luggage are identified and allowed to pass quickly, which significantly reduces the waiting time of passersby, improves security inspection efficiency, pass rate of passersby, and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0035] Figure 1 A schematic diagram of a security inspection device according to an embodiment of the present disclosure is schematically shown;

[0036] Figure 2 The flowchart of the detection method according to the embodiment of the present disclosure is schematically shown;

[0037] Figure 3 The flowchart of multi-view fusion according to an embodiment of the present disclosure is schematically shown;

[0038] Figure 4 A flowchart for identifying an object to be inspected according to an embodiment of the present disclosure is schematically shown;

[0039] Figure 5 A flowchart for identifying an object to be inspected according to another embodiment of the present disclosure is schematically shown;

[0040] Figure 6 Schematically illustrates a flow chart marked as the same object identification according to an embodiment of the present disclosure;

[0041] Figure 7 A flowchart for identifying an object to be inspected that has not been scanned by a detection device according to an embodiment of the present disclosure is schematically shown;

[0042] Figure 8 The structure block diagram of the detection device according to the embodiment of the present disclosure is schematically shown;

[0043] Fig. 9 A block diagram of an electronic device suitable for implementing the detection method according to an embodiment of the present disclosure is schematically shown.

[0044] The above drawings are associated with the following reference numerals:

[0045] 100. Security inspection equipment; 110. Detection equipment; 111. Placement area; 112. Pick-up area; 120. A channel; 121. Partition wall A; 122. Entrance access device A; 123. Exit access device A; 130. B channel; 131. Partition wall B; 132. Entrance access device B; 133. Exit access device B; 1. Camera on the right side in front of the entrance; 2. Camera on the left side in front of the entrance; 3. Camera on the left side behind; 4. Camera on the right side behind; 5. Camera on the left side in front of the exit; 6. Camera on the right side in front of the exit; 7. Identity verification device;

[0046] It should be noted that, for the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the sizes of the overall / local structures or the overall / local areas may be enlarged or reduced, that is, these drawings are not drawn according to the actual scale. DETAILED DESCRIPTION

[0047] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0048] In the technical solution of the present invention, the personnel information (including but not limited to personnel personal information, personnel image information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the personnel or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances or prompts for selection of authorization or rejection.

[0049] In scenarios such as civil aviation security checks, customs inspections, rail transit, hospital admission security checks, and inspections of special security sites, there are clear regulatory requirements for multiple angles such as biosafety, carry-on baggage safety, and nuclear radiation safety. These requirements may include one or more monitoring and inspection links such as temperature detection, health declaration verification, nuclear radiation detection, carry-on baggage inspection, and risk analysis.

[0050] Taking carry-on baggage inspection as an example, assuming that the belt speed of the radiation imaging equipment will not exceed 0.3 m / s, the length of the equipment is generally 2-3 meters. After the equipment gives an image, manual interpretation is required. Manual interpretation generally takes several seconds to more than ten seconds depending on the complexity of the image.

[0051] According to the above actual situation, it takes about 30 seconds from the time a bag is placed by a passenger to the time the bag is checked. During this time, the passenger needs to wait on the spot, which greatly affects the efficiency of equipment use and the passenger passing rate.

[0052] According to the actual situation, not every passenger will carry carry-on luggage, and passengers who do not check their carry-on luggage will need about a walking time to complete the inspection process, which is about 5-6 seconds. Obviously, in practice, passengers without luggage cannot be asked to wait, and waiting will not produce any results. Therefore, it is necessary to manually judge whether the passenger is carrying carry-on luggage. When the passenger flow is large, the number of manual workers and the speed of judgment are high, which may easily affect the efficiency of equipment use, security inspection efficiency and passenger pass rate, and may reduce the user experience.

[0053] Some embodiments of the present disclosure provide detection methods, detection devices, security inspection equipment, electronic equipment, media and program products, which can significantly improve the efficiency and user experience of the security inspection process. By using images taken from two different perspectives in the entrance area and the passage, as well as multi-perspective fused images, it is possible to quickly identify whether the passerby is carrying an object to be inspected, which not only reduces the reliance on manual judgment, reduces the risk of misjudgment and missed inspections, but also improves the accuracy and safety of security inspections. At the same time, since manual intervention is reduced, labor costs can be reduced in the long run. For example, passers-by with luggage are identified to prevent missed inspections, and passers-by without luggage are identified and allowed to pass quickly, which significantly reduces the waiting time of passers-by and improves security inspection efficiency, pass rate of passers-by and user experience.

[0054] Figure 1 The schematic diagram of the security inspection device according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown are merely examples to which the embodiments of the present disclosure may be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure may not adopt other layouts or device structures, nor does it mean that the embodiments of the present disclosure may not be used in other systems, environments or scenarios.

[0055] like Figure 1 As shown, the security inspection device 100 includes a detection device 110. The detection device 110 has a passage on at least one side for people to pass through. Figure 1The figure shows that there are passages on both sides of the detection device 110, and the objects to be inspected carried by the personnel on both sides are all subjected to radiation scanning by the detection device 110. For the passage on one side of the detection device 110, the security inspection device 100 also includes an entrance passage device and an exit passage device, and a passage is defined between the exit passage device, the entrance passage device and the partition wall. The security inspection device 100 also includes an electronic device ( Figure 1 (not shown), the electronic device can receive the image captured by the camera and the scanned image of the detection device 110, and execute the detection method provided by some embodiments of the present disclosure.

[0056] In some embodiments, the security inspection equipment 100 includes M1 first cameras (such as the right side front camera 1 at the entrance and the left side front camera 2 at the entrance), which are installed at the entrance passage device and are used to be controlled by an electronic device to photograph a first area of ​​a person, where M1 is an integer greater than or equal to 1.

[0057] In some embodiments, the security inspection equipment 100 includes M2 second cameras (such as the left side camera 3 behind the back and the right side camera 4 behind the back), which are installed at the entrance access device and are used to be controlled by the electronic device to photograph the second area of ​​the personnel, where M2 is an integer greater than or equal to 1.

[0058] In some embodiments, the security inspection equipment 100 includes M3 third cameras (such as the left side camera 5 in front of the exit and the right side camera 6 in front of the exit), which are installed in the exit passage device and are used to be controlled by the electronic device to photograph the first area of ​​the personnel, and M3 is an integer greater than or equal to 1.

[0059] Reference Figure 1 , the detection device 110 includes an A channel 120 and a B channel 130 on both sides. The entrance and exit of the A channel 120 are respectively provided with a partition wall A121, an entrance pass device A122 and an exit pass device A123. The entrance and exit of the B channel 130 are respectively provided with a partition wall B131, an entrance pass device B132 and an exit pass device B133. Taking the B channel 130 as an example, the entrance side front right camera 1, the entrance side front left camera 2, the rear left camera 3, the rear right camera 4, the exit side front left camera 5 and the exit side front right camera 6 are installed on the entrance pass device B132. The identity verification device 7, such as a face screen, a document reader and other devices, can also be installed on the entrance pass device B132. The entrance pass device or the exit pass device can be a gate machine.

[0060] In the embodiments of the present disclosure, the term "channel" refers to an area where passers-by undergo security checks. Passers-by can pass through the channel to achieve security checks on passers-by, or place objects to be inspected in the placement area 111 in the channel to achieve security checks on luggage, and passers-by can take them away from the take-up area 112. Various detection devices 110 are arranged in the channel, such as camera devices, human body security inspection devices, temperature detection devices, etc., for tracking, locating or inspecting passers-by or luggage.

[0061] As an example, the detection device 110 can be one of the security inspection devices 100 such as an X-ray security inspection machine or a CT scanning security inspection machine. It is suitable for checking whether there are prohibited items such as drugs, explosives, and flammables in the objects to be inspected such as packages, suitcases, and handbags at places such as stations, airports, and docks. The X-ray security inspection machine can scan the objects to be inspected by transmission, and the CT security inspection machine can generate a three-dimensional scanning image of the objects to be inspected by tomography, thereby realizing a rapid, three-dimensional imaging inspection of the objects to be inspected.

[0062] It should be understood that Figure 1 The number of detection devices 110, cameras and channels is merely illustrative and can be flexibly set according to actual needs.

[0063] The following will be based on Figure 1 The scene described by Figure 2~Figure 7 The detection method of the embodiment of the present disclosure is described in detail.

[0064] Figure 2 The flowchart of the detection method according to the embodiment of the present disclosure is schematically shown.

[0065] like Figure 2 As shown, this embodiment includes:

[0066] In operation S210, when a person is located at an entrance area of ​​a passage, a first area of ​​the person is photographed to obtain a first image;

[0067] When a passerby is located in the entrance area of ​​the passage, he is about to enter the passage. For example, after the person is located in the entrance area, the gate opens after face recognition and allows him to enter. The first area includes a local area of ​​the person, such as a torso area, a face area, an upper body area or a lower body area. It can be understood that the passerby can be any pedestrian to be inspected.

[0068] In operation S220, after the person passes through the entrance area and enters the passage, a second area of ​​the person is photographed to obtain a second image;

[0069] For example, after a passerby opens the gate through face recognition and enters the passage, the second area is photographed while the passerby walks forward. The second area includes a local area of ​​the passerby, such as a torso area, a face area, an upper body area, or a lower body area. The first area and the second area may partially overlap, or they may be completely independent areas.

[0070] In some embodiments, the first area includes a front area of ​​the person, and the second area includes a back area of ​​the person.

[0071] When the person is at the entrance, the image before entering the channel is captured, that is, the front area is photographed. When the person enters the channel, the image after entering the channel is captured, that is, the back area is photographed. In this way, the front and back images of the person can be obtained without the person turning around or making other movements, effectively improving the speed of security inspection.

[0072] Reference Figure 1 The B channel 130 has an entrance area provided with an entrance pass device B132, on which M1 first cameras (such as the right side camera 1 at the entrance and the left side camera 2 at the entrance) are installed, which are used to photograph the first area of ​​the person, where M1 is an integer greater than or equal to 1. The entrance pass device B132 is also equipped with M2 second cameras (such as the left side camera 3 at the back and the right side camera 4 at the back), which are used to photograph the second area of ​​the person, where M2 is an integer greater than or equal to 1.

[0073] In operation S230, multi-view fusion is performed on the first image and the second image to obtain a third image;

[0074] Multi-view fusion refers to the technology of combining images taken from different perspectives into one image. Using multiple cameras to shoot the same person from different angles, the first image and the second image are obtained, and then a third image with a more comprehensive perspective is synthesized through software algorithms.

[0075] In operation S240, the first image, the second image, and the third image are detected respectively to identify objects to be detected carried by the personnel, wherein the objects to be detected include objects to be subjected to radiation scanning by the detection device 110 beside the passage.

[0076] In operation S240 , a deep learning algorithm may be used to identify the object to be inspected in the image, for example, to identify the type, volume, and size of the object to be inspected.

[0077] Taking airport security as an example, when a passenger enters the security inspection channel (i.e., the channel), the camera at the entrance captures the first area image of the passenger. The passenger continues to move forward, and the camera in the second area captures the image after entering. These images are fused from multiple perspectives through image processing software to form a more comprehensive image. Then, the fused image is analyzed using a deep learning algorithm to identify whether the passenger is carrying luggage. If it is confirmed that the passenger is carrying luggage, the passenger should place the luggage in the placement area 111 of the detection device 110, and the detection device 110 scans the passenger's luggage in the scanning area and outputs the corresponding scanned image. Finally, the passenger leaves the security inspection channel and takes the luggage in the pick-up area 112. In the entire process, it is ensured that the scanned image, the image judgment conclusion, and the passenger who picks up the luggage are corresponding, so as to achieve continuous security inspection tracking and improve the reliability of security inspection.

[0078] According to the embodiments of the present disclosure, by using images taken from two different perspectives in the entrance area and the passage, as well as images fused from multiple perspectives, it is possible to quickly identify whether a passerby is carrying an object to be inspected, which not only reduces the reliance on manual judgment, reduces the risk of misjudgment and missed inspection, but also improves the accuracy and safety of security inspections. At the same time, since manual intervention is reduced, labor costs can be reduced in the long run. For example, passersby with luggage can be identified to prevent missed inspections, and passersby without luggage can be identified and allowed to pass quickly, which significantly reduces the waiting time of passersby, improves security inspection efficiency, pass rate of passersby, and user experience.

[0079] In some embodiments, performing multi-perspective fusion of the first image and the second image to obtain the third image includes: splicing and fusing images of persons displayed at different perspectives in the first image and the second image to obtain the third image.

[0080] In some embodiments, the first image and the second image can be input into a pre-trained attention-based generative adversarial network. First, the network uses multiple scale attention modules to extract features of the first image and the second image respectively. Then, attention maps corresponding to the first image and the second image are generated. The feature maps of the two images are combined with the corresponding attention maps and then input into the generator network to obtain a third image. In the pre-training stage, the discriminator network is used to merge the images output by the generator network, and the network parameters are updated according to the identification results.

[0081] In other embodiments, the original first image and the second image are first preprocessed to improve their quality, and then key features such as edges, textures, and color distributions are extracted. Next, the first image and the second image are aligned in the same coordinate system by image registration, and complementary information between the first image and the second image is identified by multi-view analysis. An image fusion algorithm (such as a neural network algorithm) is used to merge this information at the feature level, for example, the features of the first image and the features of the second image are converted into a common feature space to eliminate the scale and direction differences of images from different perspectives. Feature fusion is then achieved based on a fusion method based on decision rules or machine learning. Finally, the fused features are mapped back to the image space to generate a final fused image.

[0082] According to the embodiments of the present disclosure, the fused third image integrates information provided by different perspectives, which is helpful to overcome the limitation of a single perspective and improve the accuracy of identifying whether a person is carrying an object to be inspected.

[0083] Figure 3 The flowchart of multi-view fusion according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 3 As shown, this embodiment is one of the embodiments of operation S230, including:

[0085] In operation S310, N first images captured at N moments and N second images captured at N moments are obtained, where N is an integer greater than or equal to 2;

[0086] In operation S320, multi-view fusion is performed on the N first images and the N second images in a one-to-one correspondence to obtain N third images.

[0087] When a passerby is at the entrance area of ​​the passage, he is continuously photographed to obtain N first images. Correspondingly, after the passerby enters the passage, he is also continuously photographed. In this way, multiple front side images and rear side images can be obtained, and then multiple third images after multi-view fusion can be obtained.

[0088] According to the embodiments of the present disclosure, by fusing third images at multiple moments from different perspectives, passers-by can be continuously detected in time and space, effectively preventing the situation where the object to be inspected cannot be detected at a single moment or the passers-by intentionally hide the object to be inspected, and also improving the recognition accuracy.

[0089] In some embodiments, N moments form a time sequence with a chronological order. By capturing images of the time sequence, the movement of passers-by can be dynamically tracked, and the positions and states of the passers-by and the objects they carry at different time points can be recorded. Furthermore, the recognition accuracy can be improved by associating them on the timeline.

[0090] In some embodiments, for the first image and the second image fused at each moment, each image of the person is displayed from a different perspective, and the perspectives are complementary. The first image is obtained by photographing the front of the person passing by, and the second image is obtained by photographing the back of the person passing by. By fusion of the front perspective and the back perspective, a full-perspective image of the person passing by can be obtained, which helps to improve recognition ability and safety.

[0091] In some embodiments, photographing a first area of ​​a person to obtain a first image includes: photographing a first area of ​​the person to obtain a first video stream; wherein the N first images include N image frames at N moments extracted from the first video stream.

[0092] In some embodiments, photographing the second area of ​​the person to obtain the second image includes: photographing the second area of ​​the person to obtain a second video stream; wherein the N second images include N image frames at N moments extracted from the second video stream.

[0093] Reference Figure 1 , taking the entrance pass device B132 as the reference, when the person moves from a distance to the entrance area and gradually approaches, the right camera 1 and the left camera 2 in front of the entrance side continue to shoot, and combined with the rule that things are larger when they are near and smaller when they are far away, the area occupied by the person in the image gradually increases. When the person passes through the entrance pass device B132 and enters the passage, the left camera 3 and the right camera 4 behind him continue to shoot, and combined with the rule that things are larger when they are near and smaller when they are far away, the area occupied by the person in the image gradually decreases.

[0094] N image frames can be extracted from the first video stream in time sequence, and N image frames can be extracted from the second video stream in time sequence, so that the areas occupied by people in the first image and the second image to be fused are approximately the same, which is conducive to multi-perspective fusion, reduces the amount of calculation, and improves the fusion quality.

[0095] In some embodiments, for the first image and the second image to be fused, for example, an image frame at time t1 can be first extracted from the first video stream as the first image, and based on the image of the person shown in the image frame, an image frame at time t2 complementary to the person's perspective can be searched and extracted from the second video stream as the second image. Specifically, by analyzing the perspective of the image frame at time t1 and determining the required complementary perspective, image frames close to the required perspective are then screened out from the second video stream. Among them, a deep learning model, such as CNN, can be used to automatically identify the perspective of the image frame and predict the required complementary perspective.

[0096] In some embodiments, detecting the first image, the second image and the third image respectively, and identifying the object to be inspected carried by the person includes: detecting each of the N first images, the N second images and the N third images, and identifying the object to be inspected carried by the person.

[0097] Exemplarily, an image processing algorithm can be used to train a machine learning model to analyze an image to identify a specific object in the image. For example, a convolutional neural network (CNN) in deep learning is used to identify objects in an image. Models such as RCNN, Fast R-CNN, Faster R-CNN, YOLO series, AlexNet, VGGNet, GoogleLeNet, ResNet, Mask R-CNN, DeepLab, CycleGAN, StyleGAN, etc. can also be used, and the present disclosure does not specifically limit them.

[0098] In combination with the above, taking customs security inspection as an example, when the passenger advances to the entrance area, the identity verification device 7 captures the passenger's face, scans the passport information or recognizes the health declaration code, and at the same time, the right camera 1 and the left camera 2 at the entrance perform video surveillance on the front and side of the passenger. The video streams transmitted by the right camera 1 and the left camera 2 at the entrance are processed by the electronic device to capture images, which can be key frames or captures based on the image change rate. The captured images are then used to detect people and carry-on luggage through the first detection model.

[0099] When passengers enter the communication path, they will enter the field of view of the left camera 3 and the right camera 4 behind them, which perform video surveillance of the passengers from the side to the back. The video streams transmitted by the left camera 3 and the right camera 4 behind them are processed by the electronic device to capture images, which can be key frames or captures based on the image change rate. The captured images are then used to detect people and carry-on luggage through the second detection model. The first detection model or the second detection model can be deployed in the electronic device or in a server connected to the communication with the electronic device. The first detection model and the second detection model can be the same or different. The same refers to the same detection model. Different refers to the first detection model and the second detection model being different detection models pre-trained using the front image set and the back image set, respectively.

[0100] Then, the video streams of the right camera 1, the left camera 2, the left camera 3, and the right camera 4 are processed in the electronic device, and spliced ​​and fused according to different physical angles (such as different viewing angles), and a video with a time series and a third image set (including one or more third images) are obtained, wherein the video and the image show a 360° panoramic view of the passenger, and the video can be synthesized from the third image set. According to the detection and recognition of the video and the graphics, the passenger's person and carry-on luggage (common categories of carry-on luggage here include: suitcases, handbags, single-shoulder backpacks, double-shoulder backpacks, various types of handbags, etc.) can be identified. Finally, it can be obtained whether the passenger carries carry-on luggage, as well as the type and quantity of the carry-on luggage.

[0101] In this customs inspection scenario, the detection result of whether the passenger is carrying carry-on luggage can be given. Due to the continuity of video surveillance, further discussion is made on how to conduct detection in the case of continuous passage, or how to prevent the luggage carried by the passengers behind from affecting the passengers in front. In addition, for a single channel, the camera can accurately achieve one-to-one correspondence, but for a dual channel, since the detection device 110 is provided with a left channel and a right channel on both sides, and inspection and passage are carried out at the same time, and the detection device 110 has only one conveyor belt, passengers on the left and right sides can place their luggage on the conveyor belt at any time. The camera cannot analyze and determine which passenger's luggage it is, resulting in the inability to achieve a complete one-to-one association between passengers and luggage. In order to accurately bind carry-on luggage to passengers, combined with Figure 4 Further explanation.

[0102] Figure 4 The flowchart of identifying an object to be inspected according to an embodiment of the present disclosure is schematically shown.

[0103] like Figure 4 As shown, this embodiment is one of the embodiments of operation S240, including:

[0104] In operation S410, in a target image having a person and an object to be inspected, the identified person and the object to be inspected are framed to obtain a person frame and at least one object frame, wherein the target image includes at least one of the first image, the second image, and the third image;

[0105] In operation S420, at least one object to be inspected carried by the person is determined according to a spatial relationship between the person frame and each object frame.

[0106] In some embodiments, the identity of the person is bound to the object identity of each object to be inspected that the person carries.

[0107] For example, the face can be captured by the identity verification device 7, and the person can be continuously captured by the right side camera 1, the left side camera 2 or other overhead cameras, so as to obtain a stable and continuous face image or body image. The face image or body image is processed by a feature extraction algorithm, such as principal component analysis PCA, current discriminant analysis LDA, wavelet transform WaveletTransform, local binary pattern LBP or deep learning method and other algorithms, so as to obtain face feature information or body feature information, and then generate an identity, such as a face ID or body ID.

[0108] For any target image, use image recognition technology to automatically select each passenger and their luggage. For example, you can draw a rectangular frame (i.e., a person frame, here we take a rectangular frame as an example, it can also be other polygons) in the area where the human body is located to represent the spatial range of the passenger's body, and the frame is bound to the identity. Assume that the coordinates of the person frame are (x1, y1, w1, h1). In addition, when the object to be inspected is detected, the object frame (x2, y2, w2, h2) will also be marked. Among them, x, y represent the coordinates of the upper left corner of the frame on the image, and w, h represent the width and height of the rectangle.

[0109] The ownership of the luggage can be determined by analyzing the spatial relationship between the passenger and the luggage. By calculating the spatial relationship between the object frame and the person frame, for example, based on the spatial overlap relationship, if the overlap is greater than a certain ratio, it can be regarded as the passenger's carry-on luggage. At the same time, the influence of the luggage of the rear passenger on the front passenger is also excluded. Finally, the passenger's identity and its object identifier are bound in the database to facilitate luggage tracking and management.

[0110] Since carry-on luggage appears in multiple physical perspectives, it may be blocked or blocked by the passenger's body parts. The detection model may identify the same physical entity as two or more carry-on luggage, which will not only affect tracking, but also the effect of multi-perspective fusion (because if the same physical entity is identified as two or more carry-on luggage, they will not be fused into the same luggage). Figure 5 and Figure 6 Further explanation.

[0111] Figure 5 A flowchart for identifying an object to be inspected according to another embodiment of the present disclosure is schematically shown.

[0112] like Figure 5 As shown, this embodiment is one of the embodiments of operation S240, and when there are multiple target images, specifically includes:

[0113] In operation S510, each image of the object to be inspected in each target image is cropped to obtain an image set;

[0114] The image of the object to be inspected refers to a partial image clipped from the target image and containing the object to be inspected. For example, edge detection and image segmentation algorithms are used to identify and clip the image of the object to be inspected from each target image. The image set includes one or more images of the object to be inspected.

[0115] In operation S520, images in the image set that belong to the same object to be inspected are marked with the same object identifier.

[0116] Using image recognition technology, each object to be inspected in the image collection is assigned and labeled with a unique object ID. Different images of the same object to be inspected in time sequence are labeled with the same object ID and stored in the database for subsequent tracking and management.

[0117] Multiple target images can be determined from multiple first images, multiple second images, and multiple third images in a time sequence. For example, when a passerby is in the entrance area, the first image can be detected in real time and one or more target images can be determined, thereby performing frame selection and binding operations. After the passerby enters the channel, the second image can be detected in real time and one or more target images can be determined. After obtaining the first second image, a multi-view fusion operation can be performed to detect the fused third image and determine one or more target images.

[0118] According to the embodiments of the present disclosure, it is possible to automatically cut out images of the object to be inspected from multiple target images, and accurately mark these images with unique object identifiers. In addition, the unified object identifier makes the tracking of each object to be inspected more efficient and accurate.

[0119] Figure 6 A flowchart marked as the same object identification according to an embodiment of the present disclosure is schematically shown.

[0120] like Figure 6 As shown, this embodiment is one of the embodiments of operation S520, including:

[0121] In operation S610, image features are extracted from each image of the object to be inspected in the image set to obtain a feature set;

[0122] In operation S620, for any image feature in the feature set, the similarity between the image features and the remaining image features is calculated, and if the similarity is greater than or equal to a preset value, the images corresponding to the two image features are marked as the same object identifier.

[0123] In some embodiments, for any image feature in the feature set, if the similarity between it and the other image features in the current feature set is less than a preset value, a new object identifier is generated to mark the corresponding image.

[0124] Combination Figure 5 and Figure 6 For example, first, use image recognition technology to determine the carry-on baggage image and its bounding box (i.e., luggage box) related to the passenger. Then, crop the image according to these bounding boxes to obtain partial images of the carry-on baggage. Secondly, store these partial images in a suitable data structure, such as an array, list, dictionary, or queue, to form an image set. Then, from the beginning to the end of the data structure, input the partial images into the feature extraction algorithm one by one, extract the features and save them to form a carry-on baggage image feature set. There is a correlation between these features and the original image. Then, use the extracted features to compare with the existing carry-on baggage image feature set. If the similarity exceeds a preset threshold (such as 90%), they are considered to be the same physical entity and are marked with the same entity ID (i.e., object identifier).

[0125] If the feature similarity does not exceed the threshold, a new entity ID is assigned to the feature and the image and recorded in the entity ID set.

[0126] Next, check whether there are any unprocessed partial images of carry-on luggage. If so, repeat the steps of feature extraction and comparison; if not, proceed to the next step.

[0127] Afterwards, all carry-on baggage information is consolidated and exported, including passenger ID, entity ID, features, and potentially multiple images.

[0128] Finally, according to the number of generated entity IDs, the number of carry-on luggage carried by passengers can be accurately counted, thereby obtaining the total number of carry-on luggage of each passenger, so as to facilitate mutual calibration based on the number of carry-on luggage carried by passengers, the number of luggage placed in the detection device 110 for radiation scanning, and the number of luggage images output by the detection device 110.

[0129] Reference Figure 1 During the security inspection process, continuous video surveillance is carried out in the security inspection area. By shooting and tracking at the entrance and exit of the channel, a closed-loop monitoring can be formed to ensure the transparency and security of the entire security inspection process. Figure 7 Describe the detection and identification process for passers-by in the exit area.

[0130] Figure 7 The flowchart of identifying an object to be inspected that has not been irradiated and scanned by the inspection device 110 according to an embodiment of the present disclosure is schematically shown.

[0131] After the object to be inspected carried by the person is identified in operation S240, Figure 7 As shown, this embodiment includes:

[0132] In operation S710, when the person is located in the exit area of ​​the passage, the first area is photographed to obtain a fourth image;

[0133] In operation S720 , the fourth image is inspected to identify the object to be inspected that has not been irradiated and scanned by the inspection device 110 .

[0134] Reference Figure 1 An exit passage device is set in the exit area of ​​channel B 130, and M3 third cameras (left-side camera 5 in front of the exit and right-side camera 6 in front of the exit) are installed on the exit passage device to shoot the first area of ​​the personnel, where M3 is an integer greater than or equal to 1.

[0135] Exemplarily, the detection device 110 includes a placement area 111, a scanning area and a picking area 112, wherein the object to be inspected after radiation scanning is configured to be transported from the placement area 111 to the scanning area for radiation scanning, and then transported to the picking area 112; wherein the placement area 111 is located in the channel and the picking area 112 is located outside the channel.

[0136] The exit area can be the end area of ​​the passage, the area where people leave after passing through the passage. Passengers can use facial recognition to swipe open the exit gate and then leave the passage.

[0137] According to the security inspection requirements, all objects to be inspected carried by the passer-by should be placed in the detection device 110 for radiation scanning. According to the embodiment of the present disclosure, by photographing the front area of ​​the passer-by, it is possible to detect whether the passer-by is carrying an object to be inspected, because the taking area 112 is located outside the passage, and the object to be inspected carried at this time is not scanned by the detection device 110 for radiation, thereby improving the accuracy of security inspection and reducing the safety risks caused by missed inspections.

[0138] In some embodiments, after identifying the object to be inspected carried by the person, the method further includes: photographing a second area of ​​the person in the channel to obtain a fifth image, and identifying the object to be inspected that has not been radiated and scanned by the detection device 110 .

[0139] The object to be inspected carried by the identification personnel can be output before the passing personnel reach the exit area. Figure 1 In this embodiment, the security inspection scene requires that only one person passes through the passage. Then the left camera 3 and the right camera 4 behind the back can be reused to continuously shoot the rear area of ​​the passerby to detect whether the person is carrying the object to be inspected. In some embodiments, the fourth image and the fifth image can be further fused from multiple perspectives to obtain one or more sixth images.

[0140] According to the embodiments of the present disclosure, when a person is about to leave a passage, the front and rear sides of the person passing through are detected simultaneously, so that comprehensive information can be obtained and objects that have not been inspected can be accurately identified.

[0141] Combined with the above and refer to Figure 1In step 1, when the passenger enters the field of view of the right camera in front of the entrance and the left camera in front of the entrance, it is determined whether the passenger carries carry-on luggage that needs to be inspected. In addition, when the passenger passes the entrance and enters the field of view of the left camera behind and the right camera behind, it is comprehensively determined whether the passenger carries carry-on luggage that needs to be inspected by combining the multi-view fusion technology. If the passenger needs to inspect the carry-on luggage, it will continue to track, otherwise it will stop tracking and directly release it at the exit pass device. In step 2, when the passenger is in the exit area, it is determined whether the passenger carries carry-on luggage at this time. If it still carries it, the passenger has not placed the luggage that should be inspected for inspection, and then it goes to step 3, otherwise it goes to step 4. In step 3, the exit pass device will prohibit the passenger from passing and issue an alarm message to remind the staff to intervene. At the same time, the passenger's facial information (image, features, ID, etc.) and human body information (image, features, ID, etc.) will be included in the risk control system, and the subsequent camera monitoring will also identify and alarm the risk-controlled object. In step 4, here, according to the carry-on luggage inspection working mode, it is divided into two situations. For example, in the fast pass mode, there is no need to wait for the carry-on luggage inspection conclusion and it is directly released. If in the strict inspection mode, you need to wait for the carry-on baggage inspection conclusion, and will not be released directly. In step 5, if the passenger's carry-on baggage conclusion is abnormal, an alarm message will be issued to remind the staff to intervene. In the fast pass mode, the staff will catch up with the passenger to handle it. In the strict pass mode, the staff will release the passenger and the passenger will wait for processing. At the same time, the passenger's facial information (image, features, ID, etc.) and body information (image, features, ID, etc.) will be included in the risk control system, and subsequent camera monitoring will also identify and alarm the risk-controlled object. If there is no abnormality in the carry-on baggage conclusion, the passenger can pass normally.

[0142] Based on the above detection method, the present disclosure also provides a detection device. Figure 8 The device is described in detail.

[0143] Figure 8 The structural block diagram of the detection device according to the embodiment of the present disclosure is schematically shown.

[0144] like Figure 8 As shown, the detection device 800 of this embodiment includes a first shooting module 810 , a second shooting module 820 , an image fusion module 830 and an object detection module 840 .

[0145] The first photographing module 810 may perform operation S210 for photographing a first area of ​​the person to obtain a first image when the person is located at an entrance area of ​​the passage;

[0146] The second photographing module 820 may perform operation S220 for photographing a second area of ​​the person to obtain a second image after the person passes through the entrance area and enters the passage;

[0147] The image fusion module 830 may perform operation S230 to perform multi-view fusion on the first image and the second image to obtain a third image;

[0148] The object detection module 840 may perform operation S240 to detect the first image, the second image, and the third image respectively, and identify objects to be inspected carried by personnel, wherein the objects to be inspected include objects to be subjected to radiation scanning by the detection equipment beside the passage.

[0149] In some embodiments, the image fusion module 830 is used to: stitch and fuse the images of people displayed at different perspectives in the first image and the second image to obtain a third image.

[0150] In some embodiments, the image fusion module 830 is also used to: obtain N first images taken at N moments, and N second images taken at N moments, where N is an integer greater than or equal to 2; perform multi-perspective fusion on the N first images and the N second images one by one to obtain N third images.

[0151] In some embodiments, the N moments form a time sequence with a chronological order.

[0152] In some embodiments, for the first image and the second image fused at each moment, each image of the person is displayed from a different perspective, and the perspectives are complementary.

[0153] In some embodiments, the first shooting module 810 is used to shoot a first area of ​​a person to obtain a first video stream; wherein the N first images include N image frames at N moments extracted from the first video stream.

[0154] In some embodiments, the second shooting module 820 is used to shoot a second area of ​​the person to obtain a second video stream; wherein the N second images include N image frames at N moments extracted from the second video stream.

[0155] In some embodiments, the object detection module 840 is used to detect each of the N first images, the N second images, and the N third images to identify the object to be detected carried by the person.

[0156] In some embodiments, the object detection module 840 is used to select the identified persons and objects to be inspected in a target image having persons and objects to be inspected, and obtain a person frame and at least one object frame, wherein the target image includes at least one of a first image, a second image, and a third image; and determine at least one object to be inspected carried by the person based on the spatial relationship between the person frame and each object frame.

[0157] In some embodiments, the object detection module 840 is used to bind the identity of a person with the object identity of each object to be detected carried by the person.

[0158] In some embodiments, when there are multiple target images, the object detection module 840 is used to crop each image of the object to be detected in each target image to obtain an image set; and mark the images belonging to the same object to be detected in the image set as the same object identifier.

[0159] In some embodiments, marking images in an image set belonging to the same object to be inspected as the same object identification includes: extracting image features for each image of the object to be inspected in the image set to obtain a feature set; for any image feature in the feature set, calculating the similarity between it and the remaining image features, and if the similarity is greater than or equal to a preset value, marking the images corresponding to the two image features as the same object identification.

[0160] In some embodiments, for any image feature in the feature set, if the similarity between it and the other image features in the current feature set is less than a preset value, a new object identifier is generated to mark the corresponding image.

[0161] In some embodiments, after identifying the object to be inspected carried by the person, the detection device 800 also includes a third shooting module, which is used to: when the person is located in the exit area of ​​the passage, shoot the first area to obtain a fourth image; detect the fourth image to identify the object to be inspected that has not been scanned by radiation from the detection device.

[0162] In some embodiments, after identifying the object to be inspected carried by the person, the second shooting module 820 is used to: shoot a second area of ​​the person in the channel to obtain a fifth image, and identify the object to be inspected that has not been scanned by radiation from the detection device.

[0163] For the parts not mentioned in the device part, they can be understood with reference to the various embodiments of the above method. That is, the device part includes modules for executing the various steps of any one of the method embodiments described above. In addition, the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each module / unit / subunit, etc. in the device part embodiment are respectively the same or similar to the implementation methods, technical problems solved, functions implemented, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0164] According to an embodiment of the present disclosure, any multiple modules in the detection device 800 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules and implemented in one module.

[0165] According to an embodiment of the present disclosure, at least one of the detection devices 800 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the detection devices 800 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0166] Fig. 9 A block diagram of an electronic device suitable for implementing the detection method according to an embodiment of the present disclosure is schematically shown.

[0167] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0168] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0169] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0170] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0171] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0172] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0173] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 901. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0174] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0175] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0176] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0177] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0178] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0179] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A detection method, characterized in that: include: When a person is located at an entrance area of ​​a passage, photographing a first area of ​​the person to obtain a first image; After the person passes through the entrance area and enters the passage, photographing a second area of ​​the person to obtain a second image; Performing multi-view fusion on the first image and the second image to obtain a third image; The first image, the second image and the third image are detected respectively to identify the object to be detected carried by the person, wherein the object to be detected includes an object to be subjected to radiation scanning by the detection equipment beside the passage.

2. The method according to claim 1, characterized in that Performing multi-view fusion on the first image and the second image to obtain a third image includes: The images of persons displayed at different viewing angles in the first image and the second image are spliced ​​and fused to obtain the third image.

3. The method according to claim 1 or 2, characterized in that: Performing multi-view fusion on the first image and the second image to obtain a third image further includes: Obtaining N first images taken at N moments and N second images taken at N moments, where N is an integer greater than or equal to 2; The N first images and the N second images are fused in a one-to-one correspondence through multi-perspective to obtain the N third images.

4. The method according to claim 3, characterized in that The N moments form a time series with a chronological order.

5. The method according to claim 3, characterized in that: For the first image and the second image fused at each moment, each displays the person image from a different perspective, and the perspectives are complementary.

6. The method according to claim 3, characterized in that Photographing a first area of ​​the person to obtain a first image includes: photographing a first area of ​​the person to obtain a first video stream; The N first images include N image frames extracted from the first video stream at the N moments.

7. The method according to claim 3, characterized in that Photographing a second area of ​​the person to obtain a second image includes: photographing a second area of ​​the person to obtain a second video stream; The N second images include N image frames extracted from the second video stream at the N moments.

8. The method according to claim 3, characterized in that Detecting the first image, the second image, and the third image respectively to identify the object to be detected carried by the person includes: Each of the N first images, the N second images, and the N third images is detected to identify the object to be detected carried by the person.

9. The method according to any one of claims 1, 2, 4-8, characterized in that: Identifying the objects to be inspected carried by the person includes: In a target image having the person and the object to be inspected, the identified person and the object to be inspected are framed to obtain a person frame and at least one object frame, wherein the target image includes at least one of the first image, the second image, and the third image; At least one object to be inspected carried by the person is determined according to the spatial relationship between the person frame and each of the object frames.

10. The method according to claim 9, characterized in that The identity identifier of the person is bound to the object identifier of each object to be inspected that the person carries.

11. The method according to claim 9, characterized in that When there are multiple target images, identifying the object to be inspected carried by the person includes: Cut out each image of the object to be inspected in each of the target images to obtain an image set; The images in the image set belonging to the same object to be inspected are marked with the same object identifier.

12. The method according to claim 11, characterized in that Marking the images in the image set belonging to the same object to be inspected as the same object identifier includes: Extracting image features from each image of the object to be inspected in the image set to obtain a feature set; For any image feature in the feature set, the similarity between the image feature and the remaining image features is calculated. If the similarity is greater than or equal to a preset value, the images corresponding to the two image features are marked as the same object identifier.

13. The method according to claim 12, characterized in that For any image feature in the feature set, if the similarities between the remaining image features in the current feature set and the image feature are all smaller than the preset value, a new object identifier is generated to mark the corresponding image.

14. The method according to claim 1, characterized in that The entrance area is provided with an entrance passage device, and M1 first cameras are installed on the entrance passage device for photographing the first area of ​​the personnel, where M1 is an integer greater than or equal to 1.

15. The method according to claim 14, characterized in that The entrance passage device is equipped with M2 second cameras for photographing the second area of ​​the personnel, where M2 is an integer greater than or equal to 1.

16. The method according to claim 1, characterized in that After identifying the object to be inspected carried by the person, the method further includes: When the person is located in the exit area of ​​the passage, photographing the first area to obtain a fourth image; The fourth image is detected to identify the object to be detected that has not been radiated and scanned by the detection device.

17. The method according to claim 16, characterized in that After identifying the object to be inspected carried by the person, the method further includes: A second area of ​​the person in the channel is photographed to obtain a fifth image, and an object to be inspected that has not been scanned by radiation from the detection device is identified.

18. The method according to claim 16, characterized in that An exit passage device is provided in the exit area, and M3 third cameras are installed on the exit passage device for photographing the first area of ​​the personnel, where M3 is an integer greater than or equal to 1.

19. The method according to any one of claims 16 to 18, characterized in that: The detection device comprises a placement area, a scanning area and a picking area, wherein the object to be detected after radiation scanning is configured to be transported from the placement area to the scanning area for radiation scanning and then transported to the picking area; Wherein, the placement area is located inside the channel, and the taking area is located outside the channel.

20. The method according to claim 1, characterized in that The first area includes a front area of ​​the person, and the second area includes a rear area of ​​the person.

21. A detection device, characterized in that: include: A first shooting module, used for shooting a first area of ​​a person to obtain a first image when the person is located at an entrance area of ​​the passage; A second shooting module, used for shooting a second area of ​​the person to obtain a second image after the person passes through the entrance area and enters the passage; An image fusion module, used for performing multi-view fusion on the first image and the second image to obtain a third image; The object detection module is used to detect the first image, the second image and the third image respectively, and identify the object to be inspected carried by the person, wherein the object to be inspected includes an object to be subjected to radiation scanning by the detection equipment beside the channel.

22. A security inspection device, characterized in that: include: entrance access devices; an exit passage device, defining a passageway between the exit passage device and the entrance passage device; A detection device located at one side of the passage; An electronic device for implementing the detection method according to any one of claims 1 to 20.

23. The security inspection device according to claim 22, characterized in that: Also includes: M1 first cameras are installed at the entrance passage device and are used to be controlled by the electronic device to photograph a first area of ​​personnel, where M1 is an integer greater than or equal to 1.

24. The security inspection device according to claim 22, characterized in that: Also includes: M2 second cameras are installed at the entrance passage device and are used to be controlled by the electronic device to photograph a second area of ​​personnel, where M2 is an integer greater than or equal to 1.

25. The security inspection device according to claim 22, characterized in that: Also includes: M3 third cameras are installed in the exit passage device and are used to be controlled by the electronic device to photograph the first area of ​​the personnel, where M3 is an integer greater than or equal to 1.

26. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 20.

27. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 20 are implemented.

28. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 20 are implemented.

Images