Image analysis method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose an image analysis method and device, equipment and a storage medium, to solve the problem that specific objects in an image are difficult to analyze and describe in related technologies. The method comprises: obtaining a region mask image corresponding to a to-be-analyzed image; the region mask image comprises image position information and identity information of each object in the to-be-analyzed image; performing image fusion processing on the to-be-analyzed image and the region mask image to obtain a target image; inputting the target image into a pre-trained multi-modal model to perform image analysis and obtain image description information of at least one object in the to-be-analyzed image; the image description information comprises identity information and behavior information. The technical solution enables the multi-modal model to analyze and describe specific objects in an image in combination with image position information and identity information of each object, to output more targeted and valuable image description information.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image analysis method, apparatus, device and storage medium. Background Technology

[0002] With the development of computer technology, multimodal data such as images, text, and video are ubiquitous, and single-modal text or image information can no longer meet people's needs. Especially in the field of surveillance, image analysis and processing of surveillance footage can provide users with more valuable information. However, related technologies can only analyze and understand the whole image, and are powerless when it comes to detailed information about specific objects. Therefore, providing a better image analysis and processing method is particularly necessary. Summary of the Invention

[0003] The purpose of this application is to provide an image analysis method, apparatus, device, and storage medium to solve the problem in related technologies that it is difficult to perform targeted analysis and description of specific objects in images.

[0004] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows: On one hand, embodiments of this application provide an image analysis method, including: Obtain the region mask image corresponding to the image to be analyzed; the region mask image includes the image location information and identity information of each object in the image to be analyzed; The image to be analyzed and the region mask image are fused together to obtain the target image; The target image is input into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

[0005] On the other hand, embodiments of this application provide an image analysis apparatus, including: The first acquisition module is used to acquire the region mask image corresponding to the image to be analyzed; the region mask image includes the image location information and identity information of each object in the image to be analyzed; The first image fusion module is used to perform image fusion processing on the image to be analyzed and the region mask image to obtain the target image; The image analysis module is used to input the target image into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

[0006] In another aspect, embodiments of this application provide an image analysis device, including a processor; and a memory arranged to store computer-executable instructions configured to be executed by the processor to implement the image analysis method described above.

[0007] In another aspect, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a processor, implement the aforementioned image analysis method.

[0008] In another aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described image analysis method.

[0009] The technical solution of this application involves acquiring a region mask image corresponding to the image to be analyzed, and performing image fusion processing on the image to be analyzed and the region mask image to obtain a target image. Since the acquired region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the target image into a pre-trained multimodal model for image analysis, image description information of at least one object in the image to be analyzed can be obtained. This image description information includes identity information and behavioral information. It is evident that this technical solution can use a target image including the image location information and identity information of each object as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of specific objects in the image, thereby better aligning the identity information and behavioral information of each object and outputting more targeted and valuable image description information. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic flowchart of an image analysis method according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the implementation principle of the training process of a feature fusion network according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the implementation principle of a training process for a multimodal model according to an embodiment of this application; Figure 4 This is a schematic diagram of a target sample image according to an embodiment of this application; Figure 5 This is a schematic flowchart of an image analysis method according to another embodiment of this application; Figure 6 This is a schematic flowchart of an image analysis method according to another embodiment of this application; Figure 7 This is a schematic block diagram of an image analysis device according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an image analysis device according to an embodiment of this application. Detailed Implementation

[0012] This application provides an image analysis method, apparatus, device, and storage medium to solve the problem in related technologies that it is difficult to perform targeted analysis and description of specific objects in images.

[0013] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0014] Figure 1 This is a schematic flowchart of an image analysis method according to an embodiment of this application, such as... Figure 1 As shown, the method includes: S102, Obtain the region mask image corresponding to the image to be analyzed. The region mask image includes the image location information and identity information of each object in the image to be analyzed.

[0015] Optionally, the image to be analyzed can be an image of any color mode, such as an RGB image, a grayscale image, or a bitmap image; this embodiment of the application does not limit this. In the region mask image, the image position information of each object can be represented by different position coordinates, and the identity information of each object can be represented by different pixel values. For example, for the two objects Xiaoming and Xiaohong in the image to be analyzed, if pixel value 1 represents the identity information of one object as Xiaoming, and pixel value 2 represents the identity information of another object as Xiaohong, then in the region mask image, the pixel values ​​at the image position where Xiaoming is located are all 1, the pixel values ​​at the image position where Xiaohong is located are all 2, and the pixel values ​​of the background area can be set to 0. Thus, the region mask image can reflect both the image position information and the identity information of each object.

[0016] It is understood that the identity information mentioned in this embodiment includes information that can distinguish different objects, such as name, code, nationality, etc. How the specific identity information is represented in the region mask image can be preset in actual application, and this application embodiment does not limit this. Specifically, for the same type of identity information of different objects (such as the names of different objects), different objects can be represented by setting different pixel values. For different types of identity information of different objects (such as the names and nationalities of different objects), different objects can be represented by setting different types or different value ranges of pixel values, and the set of pixel values ​​corresponding to different identity information of the same object in the region mask image is represented by an identifier value. For example, pixel values ​​in numerical form can be used to represent objects with different names, and pixel values ​​in alphabetical form can be used to represent the above-mentioned objects with different nationalities. For example, if the object's name is Xiaoming and its nationality is A, the identity information of Xiaoming is represented by pixel value 1, and the nationality A is represented by pixel value A. Then, in the region mask image, the identifier value (1, A) can be used to represent the object's identity information, i.e., Xiaoming with nationality A.

[0017] S104: Perform image fusion processing on the image to be analyzed and the region mask image to obtain the target image.

[0018] The number of image channels in the target image varies depending on the image fusion processing method. In this embodiment, a suitable image fusion processing method can be selected based on the requirements of the multimodal model for the number of image channels to obtain a target image with the desired number of image channels.

[0019] S106, The target image is input into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed. The image description information includes identity information and behavioral information.

[0020] Optionally, the image analysis includes: identifying each of the at least one objects in the target image based on the identity information of at least one object, and analyzing the behavioral information of each object based on the target image.

[0021] The technical solution of this application involves acquiring a region mask image corresponding to the image to be analyzed, and performing image fusion processing on the image to be analyzed and the region mask image to obtain a target image. Since the acquired region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the target image into a pre-trained multimodal model for image analysis, image description information of at least one object in the image to be analyzed can be obtained. This image description information includes identity information and behavioral information. It is evident that this technical solution can use a target image including the image location information and identity information of each object as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of specific objects in the image, thereby better aligning the identity information and behavioral information of each object and outputting more targeted and valuable image description information.

[0022] In one embodiment, before obtaining the region mask image corresponding to the image to be analyzed (i.e., S102), the image to be analyzed can be input into a feature extraction network for feature extraction to obtain the image location information and identity information of each object in the image to be analyzed, thereby generating the region mask image corresponding to the image to be analyzed based on the image location information and identity information of each object.

[0023] The feature extraction network can be any network with feature extraction capabilities, such as the ReID (Person Re-identification) network. This application does not limit this.

[0024] In this embodiment, feature extraction is performed on the image to be analyzed to obtain the image location information and identity information of each object, thereby generating a corresponding region mask image. This makes the region mask image include the image location information and identity information of each object in the image to be analyzed, providing a data foundation for subsequent image analysis and facilitating more targeted analysis and description of the image.

[0025] In one embodiment, performing image fusion processing on the image to be analyzed and the region mask image to obtain the target image (i.e., S104) can be performed as follows: performing image stitching processing on the image to be analyzed and the region mask image to obtain the target image.

[0026] It is understandable that a region mask image is similar to a binary image. Optionally, if the image to be analyzed is a three-channel image (such as an RGB image), the target image obtained after image stitching is a four-channel image.

[0027] In this embodiment, the target image is obtained by image stitching the image to be analyzed and the region mask image. The direct stitching method directly represents the region of each object at the pixel level through the mask. This not only makes the target image, which serves as the input data for the multimodal model, carry the image location information and identity information of each object, which is beneficial for the multimodal model to perform more targeted analysis and description of the target image, but also ensures that the number of image channels of the target image meets the requirements of the multimodal model, thus ensuring that the multimodal model can accurately analyze and process the target image.

[0028] In one embodiment, image fusion processing is performed on the image to be analyzed and the region mask image to obtain the target image (i.e., S104). This can be performed by inputting the image to be analyzed and the region mask image into a feature fusion network for feature fusion to obtain the target image.

[0029] The target image and the image to be analyzed have the same number of image channels. The feature fusion network can be a network capable of fusing feature-level information, such as a convolutional network; however, this embodiment does not limit this.

[0030] In this embodiment, the input data of the feature fusion network may differ depending on the feature fusion network used. In one case, the input data of the feature fusion network consists of two images: the image to be analyzed and the region mask image mentioned above. In this case, the feature fusion network first performs image stitching on these two images to obtain a stitched image, and then performs feature fusion on the stitched image to obtain the target image. In another case, the input data of the feature fusion network consists of only one image. In this case, before inputting the image to be analyzed and the region mask image into the feature fusion network, the two images need to be stitched together to obtain a stitched image, and then this stitched image is input into the feature fusion network for feature fusion to obtain the target image.

[0031] In this embodiment, the target image is obtained by inputting the image to be analyzed and the region mask image into a feature fusion network for feature fusion. This not only ensures that the target image, which serves as input data for the multimodal model, carries the image location and identity information of each object, which is beneficial for the multimodal model to perform more targeted analysis and description of the target image, but also ensures that the number of image channels of the target image meets the requirements of the multimodal model without changing the existing model structure, thus enhancing adaptability and ensuring that the multimodal model can accurately analyze and process the target image.

[0032] In one embodiment, a feature fusion network that meets the requirements of this application embodiment can be pre-trained using sample data. As can be seen from the previous embodiment, the feature fusion network ultimately performs feature fusion processing on a stitched image. The image stitching step can be performed outside the feature fusion network or within the feature fusion network. Based on this, the feature fusion network can be trained according to the following steps A1-A2, wherein the image stitching step is performed outside the feature fusion network, such as... Figure 2 The diagram illustrates the specific training process of the feature fusion network, including: Step A1: Obtain multiple sample stitched images and the sample fusion images corresponding to each sample stitched image.

[0033] Before acquiring multiple sample stitched images, multiple second sample images can be acquired first. Next, for each second sample image, the second sample image is input into a feature extraction network for feature extraction to obtain the sample image location information and sample identity information of each sample object in the second sample image. Based on the sample image location information and sample identity information of each sample object, a sample region mask image corresponding to the second sample image is generated. Finally, the second sample image and the sample region mask image are stitched together to obtain the sample stitched image.

[0034] Step A2: Input the stitched image and the fused image of the samples into the feature fusion network to be trained for iterative training to obtain the trained feature fusion network.

[0035] In one embodiment, step A2 can be performed as follows: steps A21-A22: Step A21: For each sample stitched image, the feature fusion network to be trained is used to perform feature fusion processing on the sample stitched image to obtain the predicted fused image corresponding to the sample stitched image.

[0036] Step A22: Based on the sample fused image and the predicted fused image, iteratively adjust the network parameters of the feature fusion network until the iteration termination condition is met.

[0037] Optionally, the network parameters can be iteratively adjusted based on the difference between the sample fused image and the predicted fused image, as well as the loss function corresponding to the feature fusion network, until the iteration termination condition is met. The iteration termination condition may include reaching a preset number of iterations, and / or, the loss function converging.

[0038] In this embodiment, a feature fusion network is trained to provide a model foundation for quickly and accurately obtaining the target image based on the image to be analyzed and the region mask image.

[0039] In one embodiment, a multimodal model can be obtained by training through the following steps B1-B2, such as... Figure 3 The diagram illustrates the specific training process of a multimodal model, including: Step B1: Obtain the target sample image and the corresponding sample dataset.

[0040] The target sample image includes the sample image location information and sample identity information of each sample object within the target sample image. The sample dataset includes image description information for each sample object. The image description information includes sample identity information and sample behavior information.

[0041] For example, in the target sample image such as Figure 4 In the scenario shown, the image description information for each sample object could include: Xiaohong, wearing a red and black plaid shirt, using a mobile phone; Zhang San, wearing a blue denim shirt, holding a laptop; Li Si, wearing a plaid shirt, looking at a mobile phone; Wang Er, wearing a pink shirt, holding a laptop; and Xiaomei, wearing a blue dress, holding a laptop. It is understandable that... Figure 4 The image displayed is a grayscale image, so the clothing features of each object are not shown.

[0042] Optionally, the image description information for each sample object can also be in a question-and-answer format, following the... Figure 4 The target sample image shown can be described in a question-and-answer format with the following image description information: Q: What is everyone in the picture doing? A: There are five people in the picture, from left to right: Xiaohong, wearing a red and black plaid shirt, is using a mobile phone; Zhang San, wearing a blue denim shirt, is holding a laptop; Li Si, wearing a plaid shirt, is looking at a mobile phone; Wang Er, wearing a pink shirt, is holding a laptop; Xiaomei, wearing a blue dress, is holding a laptop. Q: Who is the person wearing the red and black plaid shirt in the picture? A: Xiaohong. Q: Where is Xiaohong in the picture? A: Xiaohong is sitting on the far left of the sofa. Q: What is Li Si doing? A: Li Si is looking at his mobile phone.

[0043] Step B2: Input the target sample image and sample dataset into the multimodal model to be trained for iterative training to obtain the trained multimodal model.

[0044] In one embodiment, step B2 can be performed as follows: steps B21-B23: Step B21: Using the multimodal model to be trained, image feature extraction processing is performed on the target sample image to obtain the image feature information of the target sample image.

[0045] The image feature information can include the sample image location information and sample identity information of each sample object. In addition, the image feature information can also include the visual information of each sample object, such as the visually observed information such as the clothing information, appearance information, and hairstyle information of each sample object.

[0046] Step B22: Based on the image feature information, generate predicted image description information for at least one sample object in the target sample image.

[0047] The predicted image description information may include predicted identity information and predicted behavior information. It is understood that the predicted image description information may also include predicted visual information.

[0048] Step B23: Based on the predicted image description information and sample dataset, iteratively adjust the model parameters of the multimodal model until the iteration termination condition is met.

[0049] Optionally, the model parameters can be iteratively adjusted based on the differences between the predicted image description information and the sample dataset, as well as the loss function corresponding to the multimodal model, until the iteration termination condition is met. The iteration termination condition may include reaching a preset number of iterations, and / or, the loss function converging.

[0050] In this embodiment, by training a multimodal model, a foundation is provided for quickly and accurately combining the image location information and identity information of each object in the input image to perform targeted analysis and description of specific objects. This better aligns the identity and behavioral information of each object, providing a more targeted and valuable image description. The trained multimodal model can already understand the image location and identity information of each object very well and can match them well. Thus, the multimodal model can generate image description information with very rich identity information. For example, previously the multimodal model could only output: "A man is watching TV." Now it can output: "Zhang San is watching TV."

[0051] In one embodiment, before obtaining the target sample image and the sample dataset corresponding to the target sample image (i.e., step B1), the target sample image can be obtained based on the first sample image through the following steps C1-C4.

[0052] Step C1: Obtain multiple first sample images.

[0053] Step C2: For each first sample image, input the first sample image into the feature extraction network for feature extraction to obtain the sample image location information and sample identity information of each sample object in the first sample image.

[0054] The feature extraction network can be any network with feature extraction capabilities, such as the ReID network, and this application embodiment does not limit it.

[0055] Step C3: Generate a sample region mask image corresponding to the first sample image based on the sample image location information and sample identity information of each sample object.

[0056] Step C4: Perform image fusion processing on the first sample image and the sample region mask image to obtain the target sample image.

[0057] The specific implementation method of obtaining the target sample image through image fusion processing is similar to the specific implementation method of obtaining the target image through image fusion processing in the above embodiments, and will not be repeated here.

[0058] In one embodiment, before obtaining the region mask image corresponding to the image to be analyzed (i.e., S102), the following steps D1-D2 can be performed: Step D1: Receive a request for behavioral information analysis of the target object. The behavioral information analysis request includes the target object's identity information.

[0059] For example, if the target is Xiaohong, a request for analyzing the target's behavioral information could be: What color clothes is Xiaohong wearing? What is Xiaohong doing? etc.

[0060] Step D2: Based on the behavioral information analysis request, obtain an image containing the target object as the image to be analyzed.

[0061] In this embodiment, the target image is input into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed (i.e., S106). This can be executed as follows: the identity information of the target object and the target image are input into the multimodal model for image analysis to obtain image description information of the target object.

[0062] Image analysis includes: identifying the target object from the target image based on the target object's identity information, and then analyzing the target object's behavioral information based on the target image.

[0063] In this embodiment, a multimodal model is used to achieve targeted search and analysis of images based on identity information. This overcomes the previous difficulty of finding out what a person has done. For example, by inputting: "Please help me find out what Zhang San did today?", the multimodal model can extract all image data containing Zhang San and analyze the corresponding behavioral information, providing the user with a very comprehensive summary.

[0064] Traditional multimodal models cannot identify the specific location of people in an image, nor do they provide identity information. Therefore, they can only analyze and understand the entire image, and they become powerless when it comes to specific regions or detailed information. Even with location and identity information, the format of location + name, such as (Zhang San: [100, 32, 564, 343]), is difficult for traditional multimodal models to understand. Therefore, multimodal models still cannot correlate specific location information with a person's identity. Coordinates are a set of discrete values ​​representing the boundaries of a region, requiring the model to calculate and infer to associate them with specific image areas. In dense scenes (e.g., multiple people standing or occlusion), the alignment of coordinates with visual features may show significant deviations. Therefore, this technical solution focuses on enabling multimodal models to better understand the relationship between identity and location. This technical solution, by embedding a mask into the original image (rather than simply inputting coordinates), can more efficiently integrate location information into a multimodal model. The mask directly labels the object's location at the pixel level, and the region information is presented in a spatial distribution form. The multimodal model can directly perceive the shape and distribution of the mask region through operations such as convolution. Each pixel value of the mask corresponds to a specific location in the original image. During the visual feature extraction stage, the multimodal model can easily associate the mask value with the visual information of the corresponding region. The pixel value of the mask region provides explicit object information (e.g., encoded as 1 to represent object 1), which can directly guide the multimodal model to focus on the target object region. Furthermore, for multimodal tasks, such as generating text descriptions or answering questions, the mask can guide the multimodal model to focus on specific regions and combine them with visual features, avoiding unnecessary background interference.

[0065] Furthermore, the image analysis method provided in this application can be applied to various image analysis scenarios. For example, it can be applied to real-time monitoring scenarios, so that the message prompts in the monitoring can be changed from "a person walked past the door" to "Zhang San walked past the door," which is more valuable. Moreover, it can provide richer guidance when multiple people are being monitored. For example, if everyone else is playing while Xiao Hong is slumped on the sofa, it can remind her, "Xiao Hong is slumped on the sofa, looks very tired and a little lonely. Should we call her to comfort her?" It can even link with other devices to make direct calls, which was impossible before the lack of identity information. It can also be applied to company monitoring scenarios, such as "Can you check when Mr. Zhang came to the company today? What clothes was he wearing?" and "Can you check where Mr. Zhang went today and what he did?" The multimodal model can focus on specific people based on identity information, generating targeted descriptions and understandings.

[0066] Figure 5 This is a schematic flowchart of an image analysis method according to another embodiment of this application, such as... Figure 5 As shown, the method includes: S501, Receive a request for behavioral information analysis of the target object, the behavioral information analysis request including the identity information of the target object.

[0067] S502, Based on the behavioral information analysis request, obtain an image containing the target object as the image to be analyzed.

[0068] S503, input the image to be analyzed into the feature extraction network for feature extraction, and obtain the image location information and identity information of each object in the image to be analyzed.

[0069] S504, based on the image location information and identity information of each object, generate the region mask image corresponding to the image to be analyzed.

[0070] S505 performs image fusion processing on the image to be analyzed and the region mask image to obtain the target image.

[0071] Optionally, the image to be analyzed and the region mask image can be stitched together to obtain the target image. Alternatively, the image to be analyzed and the region mask image can be input into a feature fusion network for feature fusion to obtain the target image; the target image and the image to be analyzed have the same number of image channels.

[0072] S506, Input the identity information and target image of the target object into the multimodal model for image analysis to obtain the image description information of the target object.

[0073] Image description information includes identity information and behavioral information. Image analysis includes: identifying the target object from the target image based on its identity information; and analyzing the behavioral information of the target object based on the target image.

[0074] The specific processes of S501-S506 described above have been explained in detail in the above embodiments and will not be repeated here.

[0075] The technical solution of this application involves acquiring an image containing the target object to be analyzed and obtaining a corresponding region mask image upon receiving a request for behavioral information analysis of the target object. Image fusion processing is then performed on the image to be analyzed and the region mask image to obtain the target image. Since the region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the identity information of the target object and the target image into a multimodal model for image analysis, image description information of the target object can be obtained, including identity information and behavioral information. It is evident that this technical solution can use the target image containing the image location information and identity information of each object, along with the target object's identity information, as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of the target object in the image, thereby better aligning the identity information and behavioral information of the target object and outputting more targeted and valuable image description information.

[0076] Figure 6 This is a schematic flowchart of an image analysis method according to another embodiment of this application, such as... Figure 6 As shown, the method includes: S601, the image to be analyzed is input into the feature extraction network for feature extraction to obtain the image location information and identity information of each object in the image to be analyzed.

[0077] S602, Generate a region mask image corresponding to the image to be analyzed based on the image location information and identity information of each object.

[0078] S603 performs image fusion processing on the image to be analyzed and the region mask image to obtain the target image.

[0079] S604, Input the target image into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed.

[0080] The image description information includes identity information and behavioral information.

[0081] The specific processes of S601-S604 described above have been explained in detail in the above embodiments and will not be repeated here.

[0082] The technical solution of this application involves acquiring a region mask image corresponding to the image to be analyzed, and performing image fusion processing on the image to be analyzed and the region mask image to obtain a target image. Since the acquired region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the target image into a pre-trained multimodal model for image analysis, image description information of at least one object in the image to be analyzed can be obtained. This image description information includes identity information and behavioral information. It is evident that this technical solution can use a target image including the image location information and identity information of each object as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of specific objects in the image, thereby better aligning the identity information and behavioral information of each object and outputting more targeted and valuable image description information.

[0083] In summary, specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing can be advantageous.

[0084] The above are the image analysis methods provided in the embodiments of this application. Based on the same idea, the embodiments of this application also provide an image analysis device.

[0085] Figure 7 This is a schematic block diagram of an image analysis apparatus according to an embodiment of this application. Please refer to... Figure 7 The image analysis device may include: The first acquisition module 710 is used to acquire the region mask image corresponding to the image to be analyzed; the region mask image includes the image location information and identity information of each object in the image to be analyzed; The first image fusion module 720 is used to perform image fusion processing on the image to be analyzed and the region mask image to obtain the target image; The image analysis module 730 is used to input the target image into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

[0086] In one embodiment, the image fusion module 720 includes: The image stitching unit is used to stitch together the image to be analyzed and the region mask image to obtain the target image.

[0087] In one embodiment, the image fusion module 720 includes: The feature fusion unit is used to input the image to be analyzed and the region mask image into the feature fusion network for feature fusion to obtain the target image; the target image and the image to be analyzed have the same number of image channels.

[0088] In one embodiment, the image analysis device further includes: The second acquisition module is used to acquire the target sample image and the corresponding sample dataset; the target sample image includes the sample image location information and sample identity information of each sample object in the target sample image; the sample dataset includes the image description information of each sample object; the image description information includes sample identity information and sample behavior information. The model training module is used to input the target sample image and sample dataset into the multimodal model to be trained for iterative training, and obtain the trained multimodal model.

[0089] In one embodiment, the image analysis device further includes: The third acquisition module is used to acquire multiple first sample images before acquiring the target sample image and the sample dataset corresponding to the target sample image; The feature extraction module is used to extract features from each first sample image by inputting the first sample image into the feature extraction network, thereby obtaining the sample image location information and sample identity information of each sample object in the first sample image. The generation module is used to generate a sample region mask image corresponding to the first sample image based on the sample image location information and sample identity information of each sample object. The second image fusion module is used to perform image fusion processing on the first sample image and the sample region mask image to obtain the target sample image.

[0090] In one embodiment, the model training module includes: The image feature extraction unit is used to perform image feature extraction processing on the target sample image through the multimodal model to be trained, so as to obtain the image feature information of the target sample image; the image feature information includes the sample image location information and sample identity information of each sample object; The generation unit is used to generate predicted image description information for at least one sample object in the target sample image based on image feature information; The iterative training unit is used to iteratively adjust the model parameters of the multimodal model based on the predicted image description information and the sample dataset until the iteration termination condition is met.

[0091] In one embodiment, the image analysis device further includes: The receiving module is used to receive a request for behavioral information analysis of the target object before acquiring the region mask image corresponding to the image to be analyzed; the behavioral information analysis request includes the identity information of the target object; The fourth acquisition module is used to acquire an image containing the target object as the image to be analyzed based on the behavioral information analysis request.

[0092] In one embodiment, the image analysis module 730 includes: The image analysis unit is used to input the identity information and target image of the target object into the multimodal model for image analysis to obtain the image description information of the target object; Image analysis includes: identifying target objects from target images based on their identity information; and analyzing the behavioral information of target objects based on target images.

[0093] The technical solution of this application involves acquiring a region mask image corresponding to the image to be analyzed, and performing image fusion processing on the image to be analyzed and the region mask image to obtain a target image. Since the acquired region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the target image into a pre-trained multimodal model for image analysis, image description information of at least one object in the image to be analyzed can be obtained. This image description information includes identity information and behavioral information. It is evident that this technical solution can use a target image including the image location information and identity information of each object as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of specific objects in the image, thereby better aligning the identity information and behavioral information of each object and outputting more targeted and valuable image description information.

[0094] Those skilled in the art will understand that Figure 7 The image analysis device described above can be used to implement the image analysis method described above. The detailed description should be similar to that in the method section above. To avoid being too complicated, it will not be repeated here.

[0095] Based on the same idea, this application also provides an image analysis device, such as... Figure 8As shown. Image analysis devices can vary considerably depending on configuration or performance, and may include one or more processors 801 and memory 802. Memory 802 may store one or more application programs or data. Memory 802 may be temporary or persistent storage. The application programs stored in memory 802 may include one or more modules (not shown), each module including a series of computer-executable instructions for the image analysis device. Furthermore, processor 801 may be configured to communicate with memory 802 and execute the series of computer-executable instructions in memory 802 on the image analysis device. The image analysis device may also include one or more power supplies 803, one or more wired or wireless network interfaces 804, one or more input / output interfaces 805, and one or more keyboards 806.

[0096] Specifically, in this embodiment, the image analysis device includes a memory and one or more application programs, wherein one or more application programs are stored in the memory, and one or more application programs may include one or more modules, and each module may include a series of computer-executable instructions for the image analysis device, and is configured to be executed by one or more processors. The one or more application programs include computer-executable instructions for performing the following: Obtain the region mask image corresponding to the image to be analyzed; the region mask image includes the image location information and identity information of each object in the image to be analyzed; The target image is obtained by performing image fusion processing on the image to be analyzed and the region mask image; The target image is input into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

[0097] The technical solution of this application involves acquiring a region mask image corresponding to the image to be analyzed, and performing image fusion processing on the image to be analyzed and the region mask image to obtain a target image. Since the acquired region mask image includes the image location information and identity information of each object in the image to be analyzed, the target image carries the image location information and identity information of each object. Therefore, by inputting the target image into a pre-trained multimodal model for image analysis, image description information of at least one object in the image to be analyzed can be obtained. This image description information includes identity information and behavioral information. It is evident that this technical solution can use a target image including the image location information and identity information of each object as input data for a multimodal model. This allows the multimodal model to combine the image location information and identity information of each object to perform targeted analysis and description of specific objects in the image, thereby better aligning the identity information and behavioral information of each object and outputting more targeted and valuable image description information.

[0098] This application also proposes a storage medium that stores one or more computer programs, each including computer-executable instructions. When executed by an electronic device including multiple applications, the computer-executable instructions enable the electronic device to perform the various processes of the above-described image analysis method embodiments and achieve the same technical effect. To avoid repetition, these will not be elaborated here.

[0099] This application also proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described image analysis method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0100] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0101] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0102] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0108] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image analysis method, characterized in that, include: Obtain the region mask image corresponding to the image to be analyzed; The region mask image includes the image location information and identity information of each object in the image to be analyzed; The image to be analyzed and the region mask image are subjected to image fusion processing to obtain a target image, which carries the image location information and identity information of each object; The target image is input into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

2. The method according to claim 1, characterized in that, The step of performing image fusion processing on the image to be analyzed and the region mask image to obtain the target image includes: The target image is obtained by stitching together the image to be analyzed and the region mask image.

3. The method according to claim 1, characterized in that, The step of performing image fusion processing on the image to be analyzed and the region mask image to obtain the target image includes: The image to be analyzed and the region mask image are input into a feature fusion network for feature fusion to obtain the target image; the target image and the image to be analyzed have the same number of image channels.

4. The method according to claim 1, characterized in that, The method further includes: Obtain a target sample image and the corresponding sample dataset; the target sample image includes sample image location information and sample identity information of each sample object in the target sample image; the sample dataset includes image description information of each sample object; the image description information includes the sample identity information and sample behavior information; The target sample image and the sample dataset are input into the multimodal model to be trained for iterative training to obtain the trained multimodal model.

5. The method according to claim 4, characterized in that, Before acquiring the target sample image and the corresponding sample dataset, the method further includes: Acquire multiple first sample images; For each of the first sample images, the first sample image is input into the feature extraction network for feature extraction to obtain the sample image location information and sample identity information of each sample object in the first sample image; Based on the sample image location information and sample identity information of each sample object, a sample region mask image corresponding to the first sample image is generated; The first sample image and the sample region mask image are subjected to image fusion processing to obtain the target sample image.

6. The method according to claim 5, characterized in that, The step of inputting the target sample image and the sample dataset into the multimodal model to be trained for iterative training to obtain the trained multimodal model includes: The target sample image is processed by the multimodal model to be trained to extract image features, thereby obtaining the image feature information of the target sample image; the image feature information includes the sample image location information and the sample identity information of each sample object; Based on the image feature information, predictive image description information for at least one sample object in the target sample image is generated; Based on the predicted image description information and the sample dataset, the model parameters of the multimodal model are iteratively adjusted until the iteration termination condition is met.

7. The method according to claim 1, characterized in that, Before obtaining the region mask image corresponding to the image to be analyzed, the method further includes: Receive a request for behavioral information analysis of a target object; the behavioral information analysis request includes the identity information of the target object; Based on the behavioral information analysis request, an image containing the target object is obtained as the image to be analyzed.

8. The method according to claim 7, characterized in that, The step of inputting the target image into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed includes: The identity information of the target object and the target image are input into the multimodal model for image analysis to obtain the image description information of the target object; The image analysis includes: identifying the target object from the target image based on the target object's identity information; and analyzing the target object's behavioral information based on the target image.

9. An image analysis device, characterized in that, include: The first acquisition module is used to acquire the region mask image corresponding to the image to be analyzed; The region mask image includes the image location information and identity information of each object in the image to be analyzed; The first image fusion module is used to perform image fusion processing on the image to be analyzed and the region mask image to obtain a target image, wherein the target image carries the image location information and identity information of each object; The image analysis module is used to input the target image into a pre-trained multimodal model for image analysis to obtain image description information of at least one object in the image to be analyzed; the image description information includes identity information and behavioral information.

10. An image analysis device, characterized in that, include: processor; as well as A memory configured to store computer-executable instructions configured to be executed by the processor to implement the image analysis method as described in any one of claims 1-8.

11. A storage medium, characterized in that, The storage medium is used to store computer-executable instructions, which, when executed by a processor, implement the image analysis method as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image analysis method according to any one of claims 1-8.

Images