Training method for fusion positioning network, storage medium and processor

Abstract

Embodiments of the present application provide a training method for a fusion positioning network, a storage medium and a processor. The method comprises: obtaining a plurality of sample image pairs in a plurality of simulation scenarios; constructing a person positioning network and an image fusion network, and training the image fusion network through the plurality of sample image pairs, wherein the image fusion network comprises a generator and a discriminator, the generator is used to generate a sample fusion image according to the sample image pairs, and a loss function of the image fusion network is used to calculate a loss value according to the sample image pairs and the sample fusion image; adjusting a weight parameter of the image fusion network according to the loss value of the loss function of the image fusion network until the loss value converges, and obtaining a trained image fusion network; bridging the generator in the trained image fusion network and the person positioning network to obtain a fusion positioning network to be trained; and training the fusion positioning network to be trained through the plurality of sample image pairs to obtain a trained fusion positioning network, thereby improving prediction accuracy.

Description

Technical Field

[0001] This application relates to the field of image recognition, and more specifically to a training method, storage medium, processor, rescue method, and wearable device for a fusion positioning network. Background Technology

[0002] Currently, networks are often trained solely using infrared images to identify people within those images. However, if people are obscured by objects or walls, the infrared images suffer from low contrast and resolution, lacking detail and edge information, making it difficult to discern the area where people are located. This results in unclear images and low accuracy. Existing technologies can fuse infrared and visible light images for person detection, but the fusion model lacks compatibility with the actual person detection model, hindering accurate predictions and detection. Summary of the Invention

[0003] The purpose of this application is to provide a training method, storage medium, processor, rescue method, and wearable device for a fusion positioning network.

[0004] To achieve the above objectives, the first aspect of this application provides a training method for a fusion localization network, comprising:

[0005] Acquire multiple sample image pairs under multiple simulated scenarios. Each sample image pair includes a sample infrared image and a sample visible light image aligned with the sample infrared image.

[0006] A personnel localization network and an image fusion network are constructed. The image fusion network is trained using multiple sample image pairs. The image fusion network includes a generator and a discriminator. The generator is used to generate a sample fusion image based on the sample image pairs. The loss function of the image fusion network is used to calculate the loss value based on the sample image pairs and the sample fusion image.

[0007] The weight parameters of the image fusion network are adjusted according to the loss value of the loss function of the image fusion network until the loss value converges, thus obtaining the trained image fusion network.

[0008] The generator and the person localization network in the trained image fusion network are bridged to obtain the fusion localization network to be trained;

[0009] The trained fusion localization network is obtained by training the network with multiple sample images.

[0010] In this embodiment of the application, before training the image fusion network using multiple sample images, the method further includes: pre-training the personnel localization network based on sample infrared images to obtain a pre-trained personnel localization network; training the image fusion network using multiple sample images includes: determining a first personnel region in the sample infrared image of the sample image pair, a first background region in the sample visible light image of the sample image pair, and a second personnel region and a second background region in the sample fusion image based on the pre-trained personnel localization network; wherein, the discriminator includes a personnel discriminator and a detail discriminator, and the loss value of the loss function of the image fusion network includes a first loss value of the loss function of the personnel discriminator, a second loss value of the loss function of the detail discriminator, and a third loss value of the loss function of the generator, wherein the first loss value is determined based on the first personnel region and the second personnel region, the second loss value is determined based on the first background region and the second background region, and the third loss value is determined based on the sample image pair and the sample fusion image.

[0011] In this embodiment of the application, bridging the generator and the person localization network in the trained image fusion network to obtain the fusion localization network to be trained includes: bridging the generator in the trained image fusion network and the pre-trained person localization network to obtain the fusion localization network to be trained.

[0012] In this embodiment, the first loss value is determined by the person discriminator based on the intensity distribution information of the first person region and the intensity distribution information of the second person region, where the intensity distribution information is determined based on the RGB values ​​of pixels between the images; the second loss value is determined by the detail discriminator based on the gradient distribution information of the first background region and the gradient distribution information of the second background region, where the gradient distribution information is determined based on the object contours in the first and second background regions; the third loss value is determined based on the first image similarity and the second image similarity, where the first image similarity is the similarity between the sample fusion image and the sample infrared image of the sample image pair, and the second image similarity is the similarity between the sample fusion image and the sample visible light image of the sample image pair.

[0013] In this embodiment, the loss function of the person discriminator is expressed as shown in formula (1):

[0014]

[0015] in, R(x) refers to the first loss value of the person discriminator, R(x) refers to the intensity data distribution of the first person region from the sample infrared image, and R(u) refers to the intensity data distribution of the second person region from the sample fused image. T (x) refers to the probability output by the person discriminator when the input sample infrared image is used. This refers to the probability output by the person discriminator when the input fused image is used. This refers to the mathematical expectation of the personnel discriminator, where the data distribution comes from the first personnel region of the sample infrared images. It refers to the mathematical expectation of the person discriminator whose data distribution comes from the second person region in the sample fusion image;

[0016] The expression for the loss function of the detail discriminator is shown in Equation (2):

[0017]

[0018] in, This refers to the second loss value of the detail discriminator. This refers to the Sobel gradient operation. This refers to the gradient map of the first background region in the visible light image of the sample after performing the Sobel gradient operation. D refers to the gradient map of the second background region of the sample fusion image after performing the Sobel gradient operation. D (x) refers to the probability output by the detail discriminator when the input sample is a visible light image. This refers to the probability output by the detail discriminator when the input fused image is used. This refers to the mathematical expectation of the detail discriminator, where the data distribution comes from the first background region of the sample visible light image. It refers to the mathematical expectation of the detail discriminator, from which the data distribution comes from the second background region in the sample visible light image;

[0019] The expression for the generator's loss function is shown in equation (3):

[0020]

[0021] in, This refers to the generator's third loss value, SSIM. u,x SSIM refers to the first image similarity between the fused sample image and the sample infrared image. u,y It refers to the second image similarity between the sample fused image and the sample visible light image.

[0022] In this embodiment of the application, training the fusion localization network to be trained using multiple sample images to obtain the trained fusion localization network includes: training the fusion localization network to be trained based on constraints to obtain the trained fusion localization network.

[0023] The expression for the constraint condition is shown in formula (4):

[0024]

[0025] in, This refers to the loss function of a pre-trained person localization network, where ψ is the weight parameter with the second weight parameter ω. d Personnel location network, u * This refers to the sample fusion image, and λ is a hyperparameter used to balance the regularization term. This refers to the regularization term in the image fusion network, where Φ represents the weight parameter ω. f Image fusion network;

[0026] The constraint conditions of the image fusion network Φ are expressed as shown in Equation (5):

[0027]

[0028] Sample fusion image u * The expression for the constraint condition is shown in formula (6):

[0029]

[0030] Among them, g T This refers to the loss function of the personnel discriminator, g T (u; x) refers to the first loss value corresponding to the fused sample image u and the sample infrared image x, g D This refers to the loss function of the detail discriminator, g D (u; y) refers to the second loss value corresponding to the sample fused image u and the sample infrared image y, f refers to the loss function of the generator, and f(u; x, y) refers to the third loss value of the generator determined by the first image similarity and the second image similarity between the sample fused image u and the sample infrared image x and the sample visible light image y, respectively.

[0031] A second aspect of this application provides a rescue method, comprising:

[0032] During the rescue operation, infrared and visible light images of the rescue location are collected in real time;

[0033] The infrared image and the aligned visible light image are input into the trained fusion localization network to obtain the target image region where the person is located in the target fusion image composed of the infrared image and the aligned visible light image;

[0034] The target fused image is sent to the display device for display, and the target image region is identified based on the target fused image.

[0035] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the aforementioned training method for a fusion localization network or the aforementioned rescue method.

[0036] The fourth aspect of this application provides a processor configured to perform the training method for a fusion positioning network described above, or configured to perform the rescue method described above.

[0037] The fifth aspect of this application provides a wearable device, comprising:

[0038] Infrared image acquisition equipment, used to acquire infrared images of an area;

[0039] Visible light image acquisition equipment, used to acquire visible light images of an area; and

[0040] The processor mentioned above.

[0041] In this embodiment of the application, the wearable device further includes:

[0042] The display module is used to display the fused image output by the trained fusion localization network, as well as the image location of the person in the fused image.

[0043] The above technical solution acquires multiple sample image pairs from various simulated scenarios, providing rich training samples for training the fusion localization network and improving the model's final test performance. A personnel localization network and an image fusion network are constructed. The image fusion network is trained using multiple sample image pairs. The weight parameters of the image fusion network are adjusted based on the loss value of its loss function until the loss value converges, resulting in a trained image fusion network. This ensures that the trained image fusion network can fully extract image features, highlighting features in both infrared and visible light images, and avoiding information loss and gradient loss. The generator in the trained image fusion network and the personnel localization network are bridged to obtain the fusion localization network to be trained. The fusion localization network to be trained is then trained using multiple sample image pairs, resulting in a trained fusion localization network. This makes the convergence of the image fusion network during training more efficient. The generator used for image fusion in the trained fusion localization network is more closely matched with the personnel localization network used for personnel detection, which is beneficial for improving personnel detection and recognition, and resulting in more accurate predictions in subsequent scenarios.

[0044] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0046] Figure 1 The schematic diagram illustrates a flowchart of a training method for a fusion localization network according to an embodiment of this application;

[0047] Figure 2 The diagram illustrates the architecture of a personnel location network according to an embodiment of this application.

[0048] Figure 3 An architectural diagram of a generator according to an embodiment of this application is illustrated schematically;

[0049] Figure 4 This schematic diagram illustrates the architecture of a discriminator according to an embodiment of this application;

[0050] Figure 5 A schematic diagram of a training image fusion network according to an embodiment of this application is shown;

[0051] Figure 6 A schematic diagram of a fused image according to an embodiment of this application is shown;

[0052] Figure 7 A schematic flowchart of a rescue method according to an embodiment of this application is shown.

[0053] Figure 8 This illustration schematically shows the identification results of a person in a target fusion image according to an embodiment of this application;

[0054] Figure 9 A schematic diagram of a smart helmet according to an embodiment of this application is shown.

[0055] Figure 10 The schematic diagram illustrates a flow chart of another rescue method according to an embodiment of this application;

[0056] Figure 11 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0058] Figure 1 A schematic flowchart illustrating a training method for a fusion localization network according to an embodiment of this application is shown. Figure 1 As shown in one embodiment of this application, a training method for a fusion localization network is provided, comprising the following steps:

[0059] Step 101: Obtain multiple sample image pairs under multiple simulated scenarios. Each sample image pair includes a sample infrared image and a sample visible light image aligned with the sample infrared image.

[0060] Step 102: Construct a personnel localization network and an image fusion network. Train the image fusion network using multiple sample image pairs. The image fusion network includes a generator and a discriminator. The generator is used to generate a sample fusion image based on the sample image pairs. The loss function of the image fusion network is used to calculate the loss value based on the sample image pairs and the sample fusion image.

[0061] Step 103: Adjust the weight parameters of the image fusion network according to the loss value of the loss function of the image fusion network until the loss value converges, and obtain the trained image fusion network.

[0062] Step 104: Bridge the generator and the person localization network in the trained image fusion network to obtain the fusion localization network to be trained.

[0063] Step 105: Train the fusion localization network to be trained using multiple sample images to obtain the trained fusion localization network.

[0064] The following description uses a processor as the method execution subject to illustrate the method provided in this application. The processor can acquire multiple sample image pairs under multiple simulated scenarios. Each sample image pair includes a sample infrared image and a sample visible light image aligned with the sample infrared image. In one embodiment, the simulated scenario includes a simulated accident location, a simulated environment, and a simulated disaster. Acquiring multiple sample image pairs under multiple simulated scenarios includes: determining multiple simulated accident locations; setting multiple simulated environments for each simulated accident location, and setting multiple simulated disasters for each simulated environment, wherein the multiple simulated disasters include at least the disaster situations of no flame and flame, flame, dense smoke, and flame and dense smoke; setting multiple sets of shooting parameters for each simulated disaster, wherein each set of shooting parameters includes at least the shooting angle and shooting height of the image acquisition device; and acquiring multiple sample image pairs under each set of shooting parameters.

[0065] For any set of shooting parameters, a sample infrared image and its corresponding sample visible light image can be aligned, and the aligned infrared and visible light images can be considered as a single sample image pair. For example, feature point matching can be used to align the infrared and visible light images. Specifically, the SIFT algorithm can be used to detect feature points in the infrared and visible light images. Then, matching can be performed based on these feature points, and the RANSAC algorithm can be used to remove mismatches. Finally, the transformation matrix can be solved using the least squares method to align the infrared and visible light images. This approach allows for flexible combination of various simulated accident locations and environments, and the construction of multiple sample image pairs for different simulated disasters at various shooting angles and heights. This provides rich training samples for training the fusion localization network, improving the model's final test performance.

[0066] The processor can build person localization networks and image fusion networks, and can train the image fusion network using multiple sample images. For example, ... Figure 2 The diagram illustrates the architecture of a personnel localization network. This network includes a feature extraction module, a feature fusion module, and an output module. The blank rectangles in the diagram represent neural networks, which include convolutional neural networks (conv), batch normalization (BN), and ReLU activation functions. The gray squares represent feature maps of different sizes, such as 80*80, 40*40, and 20*20. The feature extraction module extracts image features using components formed by multiple convolutional neural networks and activation functions. The feature fusion module obtains different levels of deep semantic features through downsampling with varying numbers of 3×3 convolutions. Then, shallow positional information is recovered from the feature maps through deconvolutional upsampling, thus fusing the deep semantic features and shallow positional features. Deep semantic features refer to information that requires knowledge to aid neural network understanding, such as face and body position. Shallow positional features include, for example, edges, colors, contours, and shapes. The output module outputs the category, bounding box, and confidence score of the bounding box through three parallel neural networks. The aforementioned personnel localization network is built based on a high- and low-order feature fusion and parallel decoupling detection scheme, which can better extract high- and low-order features.

[0067] The image fusion network includes a generator and a discriminator. The generator produces a fused image based on the sample image pairs, and the loss function of the image fusion network calculates the loss value based on the sample image pairs and the fused image. For example, ... Figure 3The diagram illustrates a generator architecture. The blank squares represent neural networks, including convolution, batch normalization (BN), and ReLU. Convolution refers to convolution, BN to batch normalization, and ReLU to the activation function. tanh is the activation function. To ensure the generator can fully extract features and has efficient inference speed, while keeping the generator lightweight, multi-level skip connections can be used to achieve the convergence of high- and low-order features throughout the network, resulting in rich image features. Specifically, after inputting sample image pairs into the generator, it can extract high-order feature information from the image through multiple subsequent neural networks. Then, it can combine this with low-order features from shallower layers to obtain multiple features in the fused image, thus avoiding information loss and gradient loss. Figure 4 The diagram illustrates the architecture of a discriminator. This discriminator comprises three ensemble blocks and one fully connected (FC) network. The FC represents a fully connected network, and the blank squares in the diagram represent the neural network, which includes convolutional neural networks (conv), batch normalization (BN), and ReLU. Convolutional neural networks (conv), batch normalization (BN), and ReLU are activation functions.

[0068] The processor can adjust the weight parameters of the image fusion network based on the loss value of its loss function until the loss value converges, resulting in a trained image fusion network. Next, the processor can bridge the generator and person localization network within the trained image fusion network to obtain a fusion localization network to be trained. The processor can then train the fusion localization network to be trained using multiple sample images, resulting in a fully trained fusion localization network.

[0069] The above technical solution acquires multiple sample image pairs from various simulated scenarios, providing rich training samples for training the fusion localization network and improving the model's final test performance. A personnel localization network and an image fusion network are constructed. The image fusion network is trained using multiple sample image pairs. The weight parameters of the image fusion network are adjusted based on the loss value of its loss function until the loss value converges, resulting in a trained image fusion network. This ensures that the trained image fusion network can fully extract image features, highlighting features in both infrared and visible light images, and avoiding information loss and gradient loss. The generator in the trained image fusion network and the personnel localization network are bridged to obtain the fusion localization network to be trained. The fusion localization network to be trained is then trained using multiple sample image pairs, resulting in a trained fusion localization network. This makes the convergence of the image fusion network during training more efficient. The generator used for image fusion in the trained fusion localization network is more closely matched with the personnel localization network used for personnel detection, which is beneficial for improving personnel detection and recognition, and resulting in more accurate predictions in subsequent scenarios.

[0070] In one embodiment, before training the image fusion network using multiple sample images, the method further includes: pre-training the personnel localization network based on sample infrared images to obtain a pre-trained personnel localization network.

[0071] Before training the image fusion network using multiple sample images, the processor can pre-train the personnel localization network based on sample infrared images to obtain a pre-trained personnel localization network. Specifically, in one embodiment, the processor can feed the sample infrared images from each sample image pair into the personnel localization network to obtain the personnel region and confidence level in each sample infrared image. Then, the processor can determine the overlap ratio between the labeled region where the personnel are located and the personnel region in each sample infrared image. Next, the processor can determine a fourth loss value of the loss function of the personnel localization network based on the confidence level and the overlap ratio. If the fourth loss value does not converge, the processor can adjust the second weight parameters of the personnel localization network based on the fourth loss value until the fourth loss value converges, thus obtaining the pre-trained personnel localization network.

[0072] In one embodiment, training an image fusion network using multiple sample image pairs includes: determining a first person region in the sample infrared image of the sample image pair, a first background region in the sample visible light image of the sample image pair, and a second person region and a second background region in the sample fusion image based on a pre-trained person localization network; wherein the discriminator includes a person discriminator and a detail discriminator, and the loss value of the loss function of the image fusion network includes a first loss value of the loss function of the person discriminator, a second loss value of the loss function of the detail discriminator, and a third loss value of the loss function of the generator, wherein the first loss value is determined based on the first person region and the second person region, the second loss value is determined based on the first background region and the second background region, and the third loss value is determined based on the sample image pair and the sample fusion image.

[0073] The processor can determine a first person region in the sample infrared image of the sample image pair, a first background region in the sample visible light image of the sample image pair, and a second person region and a second background region in the sample fused image based on a pre-trained person localization network. The discriminator includes a person discriminator and a detail discriminator. The loss function of the image fusion network includes a first loss value from the person discriminator's loss function, a second loss value from the detail discriminator's loss function, and a third loss value from the generator's loss function. The first loss value is determined based on the first and second person regions, the second loss value is determined based on the first and second background regions, and the third loss value is determined based on the sample image pair and the sample fused image.

[0074] In one embodiment, the first loss value is determined by the person discriminator based on the intensity distribution information of the first person region and the intensity distribution information of the second person region, the intensity distribution information being determined based on the RGB values ​​of pixels between the images; the second loss value is determined by the detail discriminator based on the gradient distribution information of the first background region and the gradient distribution information of the second background region, the gradient distribution information being determined based on the object contours in the first background region and the second background region; the third loss value is determined based on the first image similarity and the second image similarity, the first image similarity being the similarity between the sample fused image and the sample infrared image of the sample image pair, and the second image similarity being the similarity between the sample fused image and the sample visible light image of the sample image pair.

[0075] The network parameters of the image fusion network are adjusted alternately by the generator and discriminator. The person discriminator determines a first loss value based on the intensity distribution information of the first and second person regions. The intensity distribution information is determined based on the RGB values ​​of pixels in the images. Thus, the person discriminator guides the generator to retain the intensity distribution of the person region in the infrared image, i.e., the image brightness, when generating the fused image. The detail discriminator determines a second loss value based on the gradient distribution information of the first and second background regions. The gradient distribution information is determined based on the object contours in the first and second background regions. Thus, the detail discriminator guides the generator to retain the gradient distribution of the background region in the visible light image, i.e., the background detail texture, when generating the fused image. The generator can determine a third loss value based on the similarity between the sample fused image and the sample infrared image of the sample image pair, and the similarity between the sample fused image and the sample visible light image of the sample image pair.

[0076] In one embodiment, adjusting the weight parameters of the image fusion network based on the loss value of the loss function of the image fusion network until the loss value converges to obtain a trained image fusion network includes: determining that the trained image fusion network is obtained if the sum of the first loss value, the second loss value, and the third loss value converges; and adjusting the weight parameters of the image fusion network and re-performing the step of training the image fusion network with multiple sample images if the sum of the first loss value, the second loss value, and the third loss value does not converge, until the sum of the first loss value, the second loss value, and the third loss value converges.

[0077] If the sum of the first, second, and third loss values ​​converges, the processor can determine that the trained image fusion network has been obtained. If the sum of the first, second, and third loss values ​​does not converge, the processor can adjust the weight parameters of the image fusion network and repeat the step of training the image fusion network with multiple sample images until the sum of the first, second, and third loss values ​​converges.

[0078] like Figure 5 The diagram illustrates a training method for an image fusion network. This network includes a generator, a person discriminator, and a detail discriminator. x represents the infrared image, y represents the visible light image, and u represents the fused image. During training, the infrared image can be input into a pre-trained person localization network to obtain the foreground person in the infrared image. Since the infrared and visible light images are aligned, background texture details in the visible light image can be determined based on the foreground person and the visible light image. The infrared and visible light images can then be input into the generator to obtain the fused image. Next, the fused image can be input into the pre-trained person localization network to obtain the foreground person and background texture details in the fused image. Further, the foreground person in the infrared image and the foreground person in the fused image can be input into the person discriminator to determine whether the person is real or fake. The background texture details in the visible light image and the background texture details in the fused image can be input into the detail discriminator to determine whether the person is real or fake.

[0079] The generator can engage in adversarial training against the person discriminator (target person) and against the detail discriminator (detail texture). That is, it can determine whether the generator has successfully deceived the discriminators, making it difficult for them to distinguish between real and fake images. If the person or detail discriminator can still distinguish between real and fake images, it indicates that the fused image generated by the generator is not good enough, and the loss values ​​of the loss functions of the person and detail discriminators have not yet converged. The generator's weight parameters can be adjusted, and the image fusion network can be retrained until the generator successfully deceives both the person and detail discriminators. At this point, it can be determined that the fused image generated by the generator likely better preserves the person's position information from the infrared image and the background texture details from the visible light image, and the image fusion network training is complete.

[0080] The trained image fusion network can fuse infrared and visible light images in an image pair. For example... Figure 6The diagram illustrates a fused image. From left to right: infrared image, visible light image, and fused image. It is evident that the personnel area is more prominent in the infrared image, while the background area is more prominent in the visible light image. This allows the fused image to better preserve the positional information of the personnel and the contour details in the background area.

[0081] The aforementioned scheme suffers from the difficulty of identifying people in infrared images due to obstructions from objects or walls. Therefore, adding visible light images enhances background detail, enabling the identification of areas difficult to distinguish in the infrared image. During image fusion network training, a detail discriminator is added. This discriminator identifies gradient distribution information in the background environment, guiding the generator to preserve the texture details of the background areas in the visible light image when generating the fused image. This results in the fused image better retaining the background texture details from the visible light image. Simultaneously, a person discriminator identifies intensity distribution information in the person area, guiding the generator to preserve the intensity information of the person area in the infrared image when generating the fused image. This further enhances the preservation of the intensity distribution information from the infrared image in the fused image.

[0082] In one embodiment, bridging the generator and the person localization network in the trained image fusion network to obtain a fusion localization network to be trained includes: bridging the generator in the trained image fusion network and the pre-trained person localization network to obtain a fusion localization network to be trained.

[0083] The processor can bridge the generator in the trained image fusion network and the pre-trained person localization network to obtain the fusion localization network to be trained. Specifically, bridging the generator and the pre-trained person localization network involves adjusting the shape of the last feature map output by the generator to match the input size of the pre-trained person localization network. After that, the two networks can be directly concatenated to obtain the fusion localization network to be trained.

[0084] In one embodiment, the loss function of the person discriminator is expressed as shown in Equation (1):

[0085]

[0086] in, R(x) refers to the first loss value of the person discriminator, R(x) refers to the intensity data distribution of the first person region from the sample infrared image, and R(u) refers to the intensity data distribution of the second person region from the sample fused image. T (x) refers to the probability output by the person discriminator when the input sample infrared image is used. This refers to the probability output by the person discriminator when the input fused image is used. This refers to the mathematical expectation of the personnel discriminator, where the data distribution comes from the first personnel region of the sample infrared images. It refers to the mathematical expectation of the person discriminator whose data distribution comes from the second person region in the sample fusion image;

[0087] The expression for the loss function of the detail discriminator is shown in Equation (2):

[0088]

[0089] in, This refers to the second loss value of the detail discriminator. This refers to the Sobel gradient operation. This refers to the gradient map of the first background region in the visible light image of the sample after performing the Sobel gradient operation. D refers to the gradient map of the second background region of the sample fusion image after performing the Sobel gradient operation. D (x) refers to the probability output by the detail discriminator when the input sample is a visible light image. This refers to the probability output by the detail discriminator when the input fused image is used. This refers to the mathematical expectation of the detail discriminator, where the data distribution comes from the first background region of the sample visible light image. It refers to the mathematical expectation of the detail discriminator, from which the data distribution comes from the second background region in the sample visible light image;

[0090] The expression for the generator's loss function is shown in equation (3):

[0091]

[0092] in, This refers to the generator's third loss value, SSIM. u,x SSIM refers to the first image similarity between the fused sample image and the sample infrared image. u,y It refers to the second image similarity between the sample fused image and the sample visible light image.

[0093] Based on the above, the total loss function of the discriminator in the image fusion network is: in, The loss value is the total loss function of the discriminator. This refers to the first loss value of the personnel discriminator. This refers to the second loss value of the detail discriminator. The overall loss function of the image fusion network is:

[0094] In one embodiment, training a fusion localization network to be trained using multiple sample images to obtain a trained fusion localization network includes: training the fusion localization network to be trained based on constraints to obtain a trained fusion localization network.

[0095] The expression for the constraint condition is shown in formula (4):

[0096]

[0097] in, This refers to the loss function of a pre-trained person localization network, where ψ is the weight parameter with the second weight parameter ω. d Personnel location network, u * This refers to the sample fusion image, and λ is a hyperparameter used to balance the regularization term. This refers to the regularization term in the image fusion network, where Φ represents the weight parameter ω. f Image fusion network;

[0098] The constraint conditions of the image fusion network Φ are expressed as shown in Equation (5):

[0099]

[0100] Sample fusion image u * The expression for the constraint condition is shown in formula (6):

[0101]

[0102] Among them, g T This refers to the loss function of the personnel discriminator, g T (u; x) refers to the first loss value corresponding to the fused sample image u and the sample infrared image x, g D This refers to the loss function of the detail discriminator, g D (u; y) refers to the second loss value corresponding to the sample fused image u and the sample infrared image y, f refers to the loss function of the generator, and f(u; x, y) refers to the third loss value of the generator determined by the first image similarity and the second image similarity between the sample fused image u and the sample infrared image x and the sample visible light image y, respectively.

[0103] Among them, according to the chain rule, the weight parameter ω of the image fusion network f The loss gradient is determined by the following formula (7):

[0104]

[0105] The above constraints guide the image fusion network to find the optimal solution through the personnel localization network. This not only generates visually appealing images but also outputs detection results with better accuracy given the trained network parameters, and converges more efficiently than independent training schemes. Equation (7) above shows that the loss gradients of the personnel localization network and the image fusion network parameters are backpropagated, and the fusion loss gradient related to the fusion network parameters is also included.

[0106] Figure 1 This is a flowchart illustrating a training method for fusing a localization network in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0107] In one embodiment, such as Figure 7 As shown, a rescue method is provided, including the following steps:

[0108] Step 701: During the rescue operation, infrared and visible light images of the rescue location are collected in real time.

[0109] Step 702: Input the infrared image and the aligned visible light image into the trained fusion localization network to obtain the target image region where the person is located in the target fusion image composed of the infrared image and the aligned visible light image.

[0110] Step 703: Send the target fused image to the display device for display, and identify the target image area based on the target fused image.

[0111] During rescue operations, the processor can acquire infrared and visible light images of the rescue location in real time. For example, infrared images of the rescue location can be acquired using infrared image acquisition equipment, and visible light images can be acquired using visible light image acquisition equipment. The infrared and visible light images can then be sent to the processor separately. The infrared image acquisition equipment can be any device with image acquisition capabilities, such as an infrared camera, infrared video camera, infrared recorder, etc. The visible light image acquisition equipment can be any device with image acquisition capabilities, such as a visible light camera, visible light video camera, visible light video camera, visible light recorder, etc.

[0112] The processor can align the infrared image with the visible light image, and input the infrared image and the aligned visible light image into a trained fusion localization network to obtain the target image region where the person is located in the target fused image composed of the infrared image and the aligned visible light image. The trained fusion localization network is obtained according to the training method described above. Next, the processor can send the target fused image to a display device for display and identify the target image region based on the target fused image. The display device can be a display screen, touch screen, or LED screen, or other device with display capabilities. For example, ... Figure 8 The diagram illustrates the identification results of a person in a target fusion image. It shows the target fusion image and the target image region where the person is identified.

[0113] In one embodiment, the method further includes: determining the total number of people to be rescued in the target fused image based on the total number of areas in the target region; determining the degree of aggregation of all people to be rescued in the target fused image based on the locations of all target images and the total number of people to be rescued; and sending an assistance request to the user terminal if the degree of aggregation is greater than a preset value, so that the user terminal can dispatch additional rescue according to the assistance request.

[0114] The processor can determine the total number of people needing rescue in the target fused image based on the total number of areas in the target region, and can determine the degree of aggregation of all people needing rescue in the target fused image based on the locations of all target images and the total number of people needing rescue. If the degree of aggregation is greater than a preset value, that is, if there are many people needing rescue at the current rescue location and they are concentrated in one area, the processor can send an assistance request to the user terminal so that the user terminal can dispatch additional rescue personnel according to the assistance request.

[0115] The above technical solutions enable more accurate identification of people in need of rescue without interference from complex rescue environments. This allows for faster location of those in need, timely discovery of them, and avoidance of obstacles during the rescue process, significantly improving rescue efficiency and success rate.

[0116] Figure 7 This is a flowchart illustrating a rescue method in one embodiment. It should be understood that, although... Figure 7 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 7 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0117] In one embodiment, a storage medium is provided storing a program that, when executed by a processor, implements the training method for the fusion positioning network described above, or performs the rescue method described above.

[0118] In one embodiment, a processor is provided for running a program, wherein the program executes the training method for the fusion positioning network described above, or executes the rescue method described above.

[0119] In one embodiment, a wearable device is provided, comprising:

[0120] Infrared image acquisition equipment, used to acquire infrared images of an area;

[0121] Visible light image acquisition equipment, used to acquire visible light images of an area; and

[0122] The processor mentioned above.

[0123] Infrared image acquisition equipment can be used to acquire infrared images of an area. Infrared image acquisition equipment can be devices with image acquisition capabilities, such as infrared cameras, infrared video cameras, infrared recorders, etc. Visible light image acquisition equipment can be used to acquire visible light images of an area. Visible light image acquisition equipment can be devices with image acquisition capabilities, such as visible light cameras, visible light video cameras, visible light recorders, etc. Wearable devices can include smart helmets, smart bracelets, smartwatches, smart glasses, etc.

[0124] For example, when rescuers receive a rescue mission, they can wear smart glasses. The infrared and visible light image acquisition devices installed on the smart glasses can capture images at the position where the rescuer's head is turned. The processor can then send the target fusion image (composed of the infrared and aligned visible light images) and the personnel image area within the target fusion image to a display device. Rescuers can then rely on the personnel image area displayed on the display device to complete the rescue of people within that area. Similarly, if rescuers wear other wearable devices, the method of acquiring the personnel image area is similar to that when the wearable device is smart glasses, and will not be elaborated further here.

[0125] In one embodiment, the wearable device further includes a display module for displaying a fused image output by the trained fusion localization network, and the image location of the person in the fused image.

[0126] The display module can be used to display the fused image output by the trained fusion localization network, as well as the image location of the person within the fused image. The display module can be a device with visualization capabilities, such as a display screen or touchscreen. It provides visualized personnel detection results. In the event of a personnel rescue operation, this can further improve rescue efficiency and success rate.

[0127] In one embodiment, such as Figure 9 As shown, a smart fire helmet is provided. The smart helmet includes a power module, a visible light camera, an infrared camera, a smart analysis module, and a display screen. The power module provides power to the smart helmet. The visible light camera captures visible light images, and the infrared camera captures infrared images. The smart analysis module receives the visible light and infrared images and, based on the different resolutions of the visible light and infrared images, generates a fused image and identifies the area where the person is located within the fused image. The display module receives the fused image and identifies the area where the person is located from the fused image and displays it.

[0128] In one embodiment, such as Figure 10 As shown, a flowchart of another rescue method is provided, which is applied to situations such as... Figure 9 The smart fire helmet shown.

[0129] During rescue operations, rescuers can first turn on the power to the smart fire helmet and put it on. Then, they can acquire image data in real time via infrared and visible light cameras, i.e., infrared and visible light images. The intelligent analysis module receives the infrared and visible light images and aligns them. The intelligent analysis module can then fuse the aligned infrared and visible light images based on a target perception adversarial learning algorithm. The intelligent analysis module can then identify and locate the person to be rescued based on a personnel detection algorithm. Next, the intelligent analysis module can send the fused image to the display module and draw the person's position on the fused image. The display module can then present the fused image and the drawn personnel position to the rescuers in real time, thus completing environmental perception and the location of the person to be rescued.

[0130] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 11 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores data such as sample image pairs and target fusion images. The network interface A02 communicates with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a training or rescue method for a fusion positioning network.

[0131] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] This application provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements steps for a training method or a rescue method for a fusion positioning network.

[0133] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing a program that initializes a training method or rescue method step for a fusion positioning network.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0140] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0142] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A training method for a fusion positioning network, characterized in that, The training method comprises: obtaining a plurality of sample image pairs in a plurality of simulated scenes, each sample image pair comprising a sample infrared image and a sample visible light image aligned with the sample infrared image; constructing a person positioning network and an image fusion network, pre-training the person positioning network based on the sample infrared image, and obtaining a pre-trained person positioning network; training the image fusion network through the plurality of sample image pairs, wherein the image fusion network comprises a generator and a discriminator, the generator is configured to generate a sample fusion image according to the sample image pair, and a loss function of the image fusion network is configured to calculate a loss value according to the sample image pair and the sample fusion image; adjusting weight parameters of the image fusion network according to the loss value of the loss function of the image fusion network until the loss value converges, and obtaining a trained image fusion network; bridging the generator in the trained image fusion network and the pre-trained person positioning network to obtain a fusion positioning network to be trained; training the fusion positioning network to be trained through the plurality of sample image pairs to obtain a trained fusion positioning network; the training of the image fusion network through the plurality of sample image pairs comprises: determining a first person region of the sample infrared image in the sample image pair, a first background region of the sample visible light image in the sample image pair, a second person region of the sample fusion image, and a second background region of the sample fusion image based on the pre-trained person positioning network; wherein the discriminator comprises a person discriminator and a detail discriminator, the loss value of the loss function of the image fusion network comprises a first loss value of a loss function of the person discriminator, a second loss value of a loss function of the detail discriminator, and a third loss value of a loss function of the generator, the first loss value is determined according to the first person region and the second person region, the second loss value is determined according to the first background region and the second background region, and the third loss value is determined according to the sample image pair and the sample fusion image. 2.The training method for a fusion positioning network according to claim 1, wherein, The first loss value is determined by the person discriminator according to intensity distribution information of the first person region and intensity distribution information of the second person region, and the intensity distribution information is determined based on RGB values of pixels between images; The second loss value is determined by the detail discriminator according to gradient distribution information of the first background region and gradient distribution information of the second background region, and the gradient distribution information is determined based on object contours in the first background region and the second background region; The third loss value is determined according to a first image similarity and a second image similarity, the first image similarity is a similarity between the sample fusion image and the sample infrared image of the sample image pair, and the second image similarity is a similarity between the sample fusion image and the sample visible light image of the sample image pair. 3.The training method for a fusion positioning network according to claim 1, wherein, The expression of the loss function of the person discriminator is shown in formula (1): （1） wherein, Lperson refers to a first loss value of the person discriminator, Pperson refers to a data distribution of intensity data from a first person region of the sample infrared image, Pperson refers to a data distribution of intensity data from a second person region of the sample fusion image, Pperson refers to a probability output by the person discriminator when the sample infrared image is input, Pperson refers to a probability output by the person discriminator when the fusion image is input, Eperson refers to a mathematical expectation of the person discriminator that the data distribution comes from the first person region of the sample infrared image, Eperson refers to a mathematical expectation of the person discriminator that the data distribution comes from the second person region of the sample fusion image; An expression of a loss function of the detail discriminator is shown as formula (2): （2） wherein, denotes the second loss value of the detail discriminator, denotes the sobel gradient operation, denotes the gradient map of the first background region of the sample visible light image after the sobel gradient operation, denotes the gradient map of the second background region of the sample fusion image after the sobel gradient operation, denotes the probability output by the detail discriminator when the sample visible light image is input, denotes the probability output by the detail discriminator when the fusion image is input, denotes the mathematical expectation of the detail discriminator when the data distribution comes from the first background region of the sample visible light image, denotes the mathematical expectation of the detail discriminator when the data distribution comes from the second background region in the sample visible light image; An expression of a loss function of the generator is shown as formula (3): （3） wherein, is a third loss value of the generator, is a first image similarity between the sample fused image and the sample infrared image, is a second image similarity between the sample fused image and the sample visible light image. 4.The training method for a fusion positioning network according to claim 1, wherein, The plurality of sample images are used to train the to-be-trained fusion positioning network, and a trained fusion positioning network is obtained. The to-be-trained fusion positioning network is trained based on a constraint condition, and a trained fusion positioning network is obtained. An expression of the constraint condition is shown as formula (4): （4） wherein, denotes a loss function of the pre-trained personnel positioning network, denotes the personnel positioning network with the weight parameter being the second weight parameter , denotes a sample fused image, denotes a hyperparameter for balancing the regularization term, denotes the regularization term of the image fusion network, denotes the image fusion network with the weight parameter being the weight parameter . Image fusion network The expression of the constraint condition of the image fusion network is shown as formula (5): （5） Sample fused image The expression of the constraint condition of the sample fused image is shown as formula (6): （6） wherein, is a loss function of the person discriminator, is a first loss value corresponding to the sample fusion image u and the sample infrared image x, is a loss function of the detail discriminator, is a second loss value corresponding to the sample fusion image u and the sample infrared image y, and f is a loss function of the generator, is a third loss value of the generator determined according to a first image similarity and a second image similarity between the sample fusion image u and the sample infrared image x and the sample visible light image y, respectively.

5. A method of rescue, characterized in that, The method comprises: In the rescue operation process, real-time acquisition of infrared images and visible light images of the rescue position is performed; The infrared image and the aligned visible light image are input into the trained fusion positioning network according to the training method for the fusion positioning network according to any one of claims 1-4, to obtain a target image region in which a person is located in a target fusion image composed of the infrared image and the aligned visible light image; The target fusion image is sent to a display device for display, and the target image region is marked on the basis of the target fusion image.

6. A processor, comprising: The processor is configured to execute the training method for the fusion positioning network according to any one of claims 1-4, or execute the rescue method according to claim 5.

7. A machine-readable storage medium having stored thereon instructions, the instructions being executable by a machine to cause the machine to perform operations comprising: The instructions, when executed by the processor, cause the processor to be configured to execute the training method for the fusion positioning network according to any one of claims 1-4, or execute the rescue method according to claim 5.

8. A wearable device, comprising: Comprise: An infrared image acquisition device for acquiring infrared images of a region; A visible light image acquisition device for acquiring visible light images of the region; and The processor according to claim 6. Further comprise:

9. The wearable device of claim 8, wherein, A display module for displaying a fusion image output by the trained fusion positioning network, and an image position in which a person is located in the fusion image. ​

Images