Weight determination method and device, electronic equipment and readable storage medium

Abstract

The invention provides a weight determination method and device, electronic equipment and a readable storage medium. The method comprises the following steps: encoding position information of an object in a first image to obtain a first position encoding feature of the object; determining a first image coding feature of the first image, and extracting a second image including the object from the first image based on the first image coding feature and the first position coding feature; determining a second image coding feature of the second image, determining a text coding feature of the prompt text, and fusing the second image coding feature and the text coding feature to obtain a first fusion feature; and performing weight prediction processing on the object based on the first fusion feature to obtain the weight of the object.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a weight determination method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, its application in the animal husbandry industry has become more and more in-depth, especially in the key link of animal weighing. The intervention of artificial intelligence has greatly improved the intelligence level and accuracy of breeding management. It not only significantly reduces the intervention of manual operations and saves labor costs, but also provides farmers with detailed insights into the growth status of animals through continuous data collection and in-depth analysis, helping them to tailor personalized feeding strategies, thereby optimizing the entire breeding process and promoting the green and sustainable development of the breeding industry. Summary of the invention

[0003] The embodiments of the present application provide a weight determination method, device, electronic device and readable storage medium, which can improve the accuracy of weight prediction.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for determining weight, the method comprising:

[0006] Encoding position information of the object in the first image to obtain a first position coding feature of the object;

[0007] determining a first image coding feature of the first image, and extracting a second image including the object from the first image based on the first image coding feature and the first position coding feature;

[0008] Determine a second image coding feature of the second image, determine a text coding feature of a prompt text, and fuse the second image coding feature and the text coding feature to obtain a first fused feature, wherein the prompt text is used to indicate the weight of the generated object;

[0009] Based on the first fusion feature, weight prediction processing is performed on the object to obtain the weight of the object.

[0010] In some embodiments, the first fusion feature is obtained by a second coding model, and before encoding the position information of the object in the first image to obtain the first position coding feature of the object, the method further includes:

[0011] Performing attention processing on the image coding features of the first image sample including the object sample through the initial second coding model to obtain a seventh attention feature of the first image sample, and performing linear transformation processing on the seventh attention feature to obtain the transformed seventh attention feature;

[0012] Performing text encoding on the annotated text corresponding to the first image sample to obtain text encoding features of the annotated text;

[0013] Based on the similarity between the transformed seventh attention feature and the text encoding feature of the annotated text, the initial second encoding model is back-propagated to obtain the second encoding model.

[0014] In some embodiments, the first position coding feature is obtained by encoding through a first coding model, the first image coding feature is obtained by encoding through the first image encoder, and before encoding the position information of the object in the first image to obtain the first position coding feature of the object, the method further includes:

[0015] By using the initial first coding model, the positive sample of the image and the negative sample of the image are respectively encoded to obtain the second position coding feature of the positive sample of the image and the third position coding feature of the negative sample of the image;

[0016] Encoding the positive image sample and the negative image sample respectively through an initial first image encoder to obtain a fifth image coding feature of the positive image sample and a sixth image coding feature of the negative image sample;

[0017] determining a first similarity between the second position coding feature and the fifth image coding feature, and determining a second similarity between the third position coding feature and the sixth image coding feature;

[0018] Based on the first similarity and the second similarity, back-propagation is performed on the initial first encoding model and the initial first image encoder to obtain the first encoding model and the first image encoder.

[0019] The present application provides a weight determination device, the device comprising:

[0020] A position coding module, used for coding the position information of the object in the first image to obtain a first position coding feature of the object;

[0021] an image coding module, configured to determine a first image coding feature of the first image;

[0022] a mask generation module, configured to extract a second image including the object from the first image based on the first image coding feature and the first position coding feature;

[0023] The image coding module is further used to determine a second image coding feature of the second image;

[0024] A coding fusion module, used for determining a text coding feature of a prompt text, fusing the second image coding feature and the text coding feature to obtain a first fusion feature, wherein the prompt text is used to indicate the weight of the generated object;

[0025] A weight determination module is used to perform weight prediction processing on the object based on the first fusion feature to obtain the weight of the object.

[0026] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0027] A memory for storing computer executable instructions or computer programs;

[0028] The processor is used to implement the weight determination method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0029] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the weight determination method provided in the embodiment of the present application when executed by a processor.

[0030] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the weight determination method provided in the embodiment of the present application is implemented.

[0031] The embodiments of the present application have the following beneficial effects:

[0032] Through the first image coding features and the first position coding features, combined with the coding features and position information, the object in the first image is located more accurately, and the second image including the object is extracted from the first image, thereby improving the accuracy of object extraction, determining the text coding features of the prompt text, and fusion of the second image coding features and the text coding features to obtain the first fusion feature, thereby fully fusion of the coding features of two different types of data, effectively enhancing the richness of the feature expression of the first fusion feature, therefore, based on the feature-rich first fusion feature, the weight prediction of the object is more accurate and comprehensive, thereby improving the accuracy of weight prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1is a structural diagram of a weight determination system architecture provided in an embodiment of the present application;

[0034] Figure 2 is a schematic diagram of the structure of an electronic device for weight determination provided in an embodiment of the present application;

[0035] Figure 3A is a first flow chart of the weight determination method provided in an embodiment of the present application;

[0036] Figure 3B is a second flow chart of the weight determination method provided in an embodiment of the present application;

[0037] Figure 3C is a third flow chart of the weight determination method provided in an embodiment of the present application;

[0038] Figure 3D is a fourth flow chart of the weight determination method provided in an embodiment of the present application;

[0039] Figure 3E is a fifth flow chart of the weight determination method provided in an embodiment of the present application;

[0040] Figure 3F is a sixth flow chart of the weight determination method provided in an embodiment of the present application;

[0041] Figure 3G is a seventh flow chart of the weight determination method provided in an embodiment of the present application;

[0042] Figure 4 is a first principle diagram of the weight determination method provided in an embodiment of the present application;

[0043] Figure 5 is a second principle diagram of the weight determination method provided in an embodiment of the present application;

[0044] Figure 6 is a third principle diagram of the weight determination method provided in an embodiment of the present application;

[0045] Figure 7 is a fourth principle diagram of the weight determination method provided in an embodiment of the present application;

[0046] Figure 8 is a fifth principle diagram of the weight determination method provided in an embodiment of the present application;

[0047] Fig. 9 is a sixth principle diagram of the weight determination method provided in an embodiment of the present application;

[0048] Fig.10 This is the seventh principle diagram of the weight determination method provided in the embodiment of the present application.

[0049] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0051] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Patch: A smaller, regularly shaped sub-region extracted from an image, usually a square or rectangle. In the image processing process, image patches are used to extract local features. Since these small patches contain local information of the image, they can help neural networks recognize and learn image features such as texture, edges, and colors.

[0058] 2) Condition: A condition refers to restrictions, rules, or guidelines imposed on data, models, or algorithm behavior in a specific context. For example, in a classification task, a condition may restrict the model to only recognize images of a specific category. In a recommendation system, based on the user's historical behavior, some conditional rules can be set to recommend items that may be of interest.

[0059] 3) Mask Image: In the field of computer vision, a mask image is a special image, usually a binary image, used to indicate whether the pixels in a specific area of ​​the original image belong to a specific category or object. Each pixel value in the mask image is usually 0 or 1 (sometimes other encoding methods are also used, such as -1 for background and 1 for foreground object), where 0 indicates that the pixel does not belong to the area of ​​interest, and 1 (or other specific values) indicates that the pixel belongs to the area of ​​interest.

[0060] 4) Contrastive Learning: It is an unsupervised learning method. Its core idea is to learn the representation of data by comparing the similarities and differences between positive and negative samples. Positive samples usually refer to samples of the same category or sample transformations obtained through data enhancement, and negative samples usually refer to samples of different categories. They should be as far away from positive samples as possible in the feature space.

[0061] 5) Large Language Models (LLM): refers to pre-trained language models with a very large number of parameters. The number of parameters of large language models usually reaches billions or even hundreds of billions, which enables them to capture complex patterns and subtle differences in language. They can understand and generate human language through unsupervised learning on large text corpora.

[0062] 6) Prompt: refers to the input text used to guide the model to generate text or perform specific tasks. The prompt usually contains some information about the expected output. It can help the model understand the context and, in some cases, guide the model to generate a response that meets specific requirements.

[0063] Embodiments of the present application provide a weight determination method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of weight prediction.

[0064] The weight determination method provided in the embodiment of the present application can be implemented independently by the terminal or the server; it can also be implemented jointly by the terminal and the server. For example, the terminal independently undertakes the weight determination method described below, or the terminal sends a weight determination request containing a first image and position information to the server, and the server executes the weight determination method based on the received weight determination request containing the first image and position information.

[0065] The electronic device for weight determination provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present application does not limit this.

[0066] See also Figure 1 , Figure 1 It is a structural diagram of the weight determination system architecture provided in the embodiment of the present application. In the weight determination system 10 provided in the embodiment of the present application, in order to support a weight determination application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0067] Terminal 400 may be used to obtain a weight determination request including a first image and position information.

[0068] In some embodiments, a weight determination plug-in may be implanted in the client running in the terminal 400 to implement a weight determination method locally on the client. For example, the terminal 400 calls the weight determination plug-in to implement the weight determination method, extracts a second image including an object from the first image through the first image coding feature and the position coding feature, determines the second image coding feature of the second image and the text coding feature of the prompt text, fuses the second image coding feature and the text coding feature to obtain a first fused feature, and performs weight prediction processing on the object based on the first fused feature to obtain the weight of the object.

[0069] In some embodiments, after the terminal 400 obtains a weight determination request including a first image and position information, the weight determination interface of the server 200 is called (which can be provided in the form of a cloud service). The server 200 implements a weight determination method through a weight determination plug-in, extracts a second image including an object from the first image through the first image coding feature and the position coding feature, determines the second image coding feature of the second image and the text coding feature of the prompt text, fuses the second image coding feature and the text coding feature to obtain a first fusion feature, performs weight prediction processing on the object based on the first fusion feature to obtain the weight of the object, and returns the weight of the object to the terminal 400 to display the weight of the object in the terminal 400.

[0070] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for weight determination provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1 The terminal 400 or the server 200 in the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0071] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0072] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0073] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0074] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0075] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0076] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0077] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0078] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0079] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The weight determination device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a position encoding module 5551, an image encoding module 5552, a mask generation module 5553, a coding fusion module 5554 and a weight determination module 5555. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0080] As mentioned above, the electronic device implementing the weight determination method of the embodiment of the present application can be a terminal, a server, or a combination of the two, so the execution subject of each step will not be repeated below, see Figure 3A , Figure 3A is a first flow chart of the weight determination method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0081] In step 101, position information of an object in a first image is encoded to obtain a first position coding feature of the object.

[0082] As an example, based on the first coding model, the position information of the object in the first image is encoded to obtain the first position coding feature of the object, wherein the position information is used to indicate the position of the object in the first image, such as the coordinate information of the geometric center point of the object in the first image, and the object refers to a target object in the first image. For example, in the chicken weight estimation application scenario, the object may be a chicken target, the first image may be an image of a chicken farm, and the position information of the object in the first image may be the coordinate point (x, y).

[0083] In some embodiments, the position information can be obtained through the following steps: obtaining a bounding box of the object, determining the coordinate information of the center point of the bounding box based on the coordinate information of the upper left corner and the coordinate information of the upper right corner of the bounding box, using the coordinate information of the center point of the bounding box as the position information of the object in the first image, encoding the position information, and obtaining a first position coding feature, wherein the bounding box of the object can be obtained by performing object recognition processing on the first image using a trained position detection model, which is not limited here.

[0084] In some embodiments, Figure 3A The illustrated step 101 can be implemented by the following steps: mapping the position information to obtain mapping features of the position information; performing sequence alignment on the mapping features of the position information to obtain aligned mapping features; performing linear transformation on the aligned mapping features to obtain a first position coding feature.

[0085] As an example, the location information (x, y) is processed with a point hint, and one-dimensional data l is added to the location information to identify the relationship between the location information and the object, so as to obtain the location information (x, y, l) after the point hint processing. Based on the nonlinear activation function (1.1), the location information after the point hint processing is mapped to obtain the mapping feature of the location information. Formula (1.1) is as follows:

[0086] v = g(w T *(x,y,l)+b) (1.1)

[0087] Among them, v is the mapping feature of the location information, C p is a hyperparameter, g is a nonlinear activation function, and w T is the weight value of the nonlinear activation function (weights), b is the bias of the nonlinear activation function, b∈R 1*256, l is used to identify the relationship between the location information and the object. In the embodiment of the present application, l=1, and R is an arbitrary real number.

[0088] Through sequence alignment processing, the feature dimension of the mapping feature of the position information is expanded to obtain the mapping feature after sequence alignment, and the mapping feature after sequence alignment is linearly transformed to obtain the first position coding feature, wherein the sequence alignment processing includes edge filling processing and mask processing on the mapping feature of the position information, and the feature dimension of the first position coding feature is N*256, where N is the number of image blocks obtained after the first image is split below.

[0089] In the related art, the first encoding model based on the Transformer architecture has a large amount of computation and slow calculation. The implementation of this application provides a lightweight universal first encoding model, which significantly reduces the computational complexity while ensuring performance, making the encoding process more efficient and meeting the needs of real-time processing.

[0090] In step 102, a first image coding feature of a first image is determined, and based on the first image coding feature and a first position coding feature, a second image including an object is extracted from the first image.

[0091] As an example, based on a first image encoder, image encoding processing is performed on the first image to obtain a first image coding feature of the first image, based on a mask generator, the first image coding feature and the first position coding feature are processed to obtain a mask feature, and then based on the mask feature, object extraction processing is performed on the object included in the first image to obtain a second image including the object, wherein the second image is a cutout image including the object.

[0092] For example, in the application scenario of chicken weight estimation, the first image may be an image of a chicken farm, which includes multiple chicken targets, the object may be chicken target A, one of the multiple chicken targets included in the chicken farm image, and the second image is a cutout image of chicken target A; or, in the application scenario of express weight estimation, the first image may be an image of an express transfer station, which includes multiple package targets, the object may be package target B, one of the multiple package targets included in the express transfer station image, and the second image is a cutout image of package target B.

[0093] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the weight determination method provided in an embodiment of the present application, Figure 3A The "determining the first image coding feature of the first image" in step 102 shown can be implemented through the following steps 1021 to 1024, which are described in detail below.

[0094] In step 1021, the first image is encoded to obtain a first encoding feature of the first image.

[0095] In some embodiments, see Figure 3C , Figure 3C is a third flow chart of the weight determination method provided in the embodiment of the present application, Figure 3B The illustrated step 1021 can be implemented by following the steps 201 to 203, which are described in detail below.

[0096] In step 201, a first image is split to obtain a plurality of image blocks of the first image.

[0097] As an example, the first image is split to obtain a plurality of image blocks of the first image, each image block having the same size.

[0098] For example, the first image is represented as I, I∈R H*W*C , where H is an arbitrary real number, H is the height of the first image, W is the width of the first image, and C is the number of channels of the first image. The first image I is split and the split first image is represented as I′, I′∈R N*D , where N is the number of image blocks, and the size of the image block is represented by p*p, D is the feature dimension of each image block, D = p 2 *Each row of data in C, I′ corresponds to a one-dimensional vector representation of an image block.

[0099] In step 202, image encoding is performed on each image block to obtain a second encoding feature of each image block.

[0100] As an example, the following processing is performed on each image block: determine the one-dimensional vector representation of the image block, perform attention processing on the one-dimensional vector representation of the image block based on the attention layer processing formula (1.2), and obtain the second encoding feature of the image block. Formula (1.2) is as follows:

[0101]

[0102] Among them, O i is the second encoding feature of the i-th image block among the N image blocks, O i∈R 1*D , D is the feature dimension of each image block, R is an arbitrary real number, and the query value Q i =I′ i *W Q , key value K j =I′ j *W K , value V i =I′ i *WV , W Q is the weight corresponding to the query value, W K is the weight corresponding to the key value, W V is the weight corresponding to the value, W Q , W K , W V ∈R D*f , f is a hyperparameter, I′ i is the i-th row data of the first image I′ after splitting, which can also be represented as a one-dimensional vector representation of the i-th image block. Sim is the distance metric function (1.3). Formula (1.3) is as follows:

[0103] Sim(Q,K)=ReLU(Q)*ReLU(K) T (1.3)

[0104] Among them, ReLU is a nonlinear activation function.

[0105] In order to facilitate the explanation of the scheme, the image encoding of the image block is implemented through a layer of attention layer. In some embodiments, the second encoding feature of the image block can also be obtained by performing image encoding on the image block based on M cascaded attention layers. Based on M cascaded attention layers, image encoding of the image block can be implemented through the following steps: determining a one-dimensional vector representation of the image block; based on the first attention layer, performing attention processing on the one-dimensional vector representation of the image block to obtain the output feature of the first attention layer; based on the mth attention layer, performing attention processing on the output feature of the m-1th attention layer to obtain the output feature of the mth attention layer; based on the Mth attention layer, performing attention processing on the output feature of the M-1th attention layer to obtain the second encoding feature of the image block, 1<m<M.

[0106] In step 203, a plurality of second coding features are spliced ​​to obtain a first coding feature of the first image.

[0107] As an example, based on the vertical stacking method, the N second coding features of the N image blocks are spliced ​​to obtain the first coding feature O of the first image, O∈R N*D , where D is the feature dimension of each image block, R is an arbitrary real number, and B is the number of image blocks.

[0108] In step 1022, convolution processing is performed on the first image to obtain convolution features of the first image.

[0109] As an example, edge filling processing is performed on the first image I to obtain the first image I″ after edge filling processing, I″∈R F*F*D, perform deep convolution on the edge-filled image I″ to obtain the convolution feature O′ of the first image, O′∈R 1*D , where the size of the convolution kernel in the deep convolution process is F*F, F is a hyperparameter, D is the feature dimension of each image block, and R is an arbitrary real number.

[0110] In step 1023, the first coding feature and the convolution feature are fused to obtain a second fused feature.

[0111] As an example, the fusion processing of the first coding feature and the convolution feature may be matrix stacking fusion of the first coding feature and the convolution feature, or matrix addition fusion of the first coding feature and the convolution feature, or the first coding feature and the convolution feature are fused based on the attention mechanism, which is not limited here. Taking matrix stacking fusion as an example, based on the vertical stacking method, the first coding feature O and the convolution feature O are fused. ′ Perform fusion processing to obtain the second fusion feature T o , T o ∈R (N+1)*D , where D is the feature dimension of each image block, R is an arbitrary real number, and N is the number of image blocks.

[0112] In step 1024, a linear transformation is performed on the second fused feature to obtain a first image coding feature.

[0113] As an example, based on the linear regression layer, the second fusion feature is dimensionally transformed to obtain the first image coding feature after dimension transformation, wherein the feature dimension of the first image coding feature is N*256, and N is the number of image blocks.

[0114] In an embodiment of the present application, the first image encoder can not only more accurately depict the local features of the image, but also effectively capture the detailed information of the image by encoding the first image after splitting and rearrangement, so as to form a closer and more intuitive corresponding matching relationship with the position coding in the subsequent process. In addition, the image encoder in the related art has a huge number of parameters and slow calculation. The implementation of the present application provides a lightweight general image encoder, which greatly reduces the number of parameters while maintaining the necessary performance, thereby significantly improving the calculation efficiency, making the image encoding process faster, and reducing resource consumption, providing a more efficient and practical solution for the implementation of image processing tasks in various application scenarios.

[0115] In some embodiments, see Figure 3D , Figure 3D is a fourth flow chart of the weight determination method provided in an embodiment of the present application, Figure 3AThe step 102 of "extracting a second image including an object from the first image based on the first image coding feature and the first position coding feature" can be implemented by following the steps 1025 to 1028, which are described in detail below.

[0116] In step 1025, a first attention processing is performed on the first image coding feature and the first position coding feature to obtain a first attention feature.

[0117] As an example, based on the first position coding feature, the first image coding feature is cross-attention processed to obtain the first attention feature, wherein the cross-attention processing formula (1.4) of the first attention processing is as follows:

[0118]

[0119] in, is the first attention feature O 1 The i-th row data, O 1 ∈R N*512 , I is the first image coding feature, C is the first position coding feature, and the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, I i is the i-th row data of the first image encoding feature, C i is the i-th row data of the first position coding feature, Sim is the distance metric function, and the distance metric function formula refers to the formula (1.3) shown above.

[0120] In step 1026, a second attention process is performed on the first image coding feature and the first position coding feature to obtain a second attention feature.

[0121] As an example, the second attention processing includes cross-attention processing and feature extraction processing.

[0122] In some embodiments, Figure 3D The shown step 1026 can be implemented by the following steps: based on the first image coding feature, cross-attention processing is performed on the first position coding feature to obtain a fourth attention feature; and feature extraction processing is performed on the fourth attention feature to obtain a second attention feature.

[0123] As an example, the cross attention processing formula (1.5) for the second attention processing is as follows:

[0124]

[0125] in, The fourth attention feature O 2 The i-th row data, O 2 ∈R N*256 , I is the first image coding feature, C is the first position coding feature, and the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, I i is the i-th row data of the first image encoding feature, C i is the i-th row data of the first position coding feature, Sim is the distance metric function, and the distance metric function formula refers to the formula (1.3) shown above.

[0126] Based on the first linear layer, the fourth attention feature is extracted to obtain the second attention feature O 3 , O 3 ∈R N*512 , where the first linear layer includes learnable parameters.

[0127] In step 1027, the first attention feature and the second attention feature are subjected to third attention processing to obtain a third attention feature.

[0128] As an example, the formula (1.6) of the third attention process is as follows:

[0129]

[0130] in, is the third attention feature O 4 The i-th row data, O 4 ∈R N*1024 , O 1 is the first attention feature, O 3 is the second attention feature, and the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, is the i-th row data of the first attention feature, is the i-th row data of the second attention feature, Sim is the distance measurement function, and the distance measurement function formula can be found in formula (1.3) shown above.

[0131] In step 1028, a second image is determined from the first image based on the third attention feature.

[0132] In some embodiments, Figure 3D The shown step 1028 can be implemented by the following steps: performing feature extraction processing on the third attention feature to obtain the fifth attention feature; performing upsampling processing on the fifth attention feature to obtain a mask feature; based on the mask feature, performing object extraction processing on the object included in the first image to obtain a second image including the object.

[0133] As an example, based on the second linear layer, the third attention feature is subjected to feature extraction processing to obtain the fifth attention feature O 5 , O 5 ∈R N*2048 , upsample the fifth attention feature to obtain the mask feature O 6 , O 6 ∈R H*W*1 , where the second linear layer includes learnable parameters.

[0134] The mask feature includes a first element and a second element. The area where the first element in the mask feature is located is the area where the object is located in the first image. The area where the second element in the mask feature is located is the area where the object does not exist in the first image. The elements of the area where the first element in the first image coding feature is located are extracted to obtain a second image including the object.

[0135] In an embodiment of the present application, the mask generator uses the first position encoding feature as a condition to specify region-level prior information for extracting a second image including an object from the first image, thereby improving the accuracy of mask feature generation, so that the mask can more accurately depict the area of ​​the target object, thereby optimizing the performance of visual tasks such as image segmentation and target detection.

[0136] In step 103, the second image coding feature of the second image is determined, and the text coding feature of the prompt text is determined, and the second image coding feature and the text coding feature are fused to obtain a first fused feature.

[0137] As an example, based on the second image encoder, the second image coding feature of the second image is determined, and based on the second coding model, the second image coding feature and the text coding feature are fused to obtain a first fused feature, wherein the prompt text is used to indicate the weight of the generated object, for example, the prompt text may be "How much does this chicken weigh?" The text coding feature may be obtained by performing text encoding processing on the prompt text by a trained text encoder, the feature dimension of the text coding feature is K*512, where K is a hyperparameter, and the text encoder may use the Word2Vec model, the bidirectional encoder of the transformer (BERT, Bidirectional Encoder Representations from Transformers) or other text coding models, which are not limited here.

[0138] In some embodiments, the first image encoding feature is obtained by encoding the first image by a first image encoder. Figure 3A The "determining the second image coding feature of the second image" in the shown step 103 can be achieved by the following steps: encoding the second image by a first image encoder to obtain a third image coding feature, and encoding the second image by a second image encoder to obtain a fourth image coding feature; normalizing the fourth image coding feature to obtain a normalized fourth image coding feature; and performing a feature transformation on the third image coding feature based on the normalized fourth image coding feature to obtain a second image coding feature.

[0139] As an example, a second image coding feature of a second image is determined by querying an image encoder, where the query image encoder includes a first image encoder and a second image encoder. When training the query image encoder, the parameters of the first image encoder may be frozen and only the parameters of the second image encoder may be updated.

[0140] Among them, the model structures of the first image encoder and the second image encoder can be consistent, the model parameters of the first image encoder and the second image encoder are different, the training directions of the first image encoder and the second image encoder are different, the first image encoder is used to extract overall features in the image, and the second image encoder is used to extract high-value features in the image.

[0141] For detailed steps of encoding the second image by the first image encoder to obtain the third image coding feature, and encoding the second image by the second image encoder to obtain the fourth image coding feature, please refer to steps 1021 to 1024 above, which are not described in detail here. The feature dimensions of the third image coding feature and the fourth image coding feature are both N*256. The fourth image coding feature is normalized to obtain the normalized fourth image coding feature, wherein the normalization of the fourth image coding feature may be based on a normalized exponential function (softmax function), the fourth image coding feature is processed, or other normalization processing methods are used, which are not limited here. Based on the normalized fourth image coding feature, the third image coding feature is feature transformed to obtain the second image coding feature, and the second image coding feature is characterized as O 7 , O 7 ∈R N*256 .

[0142] In an embodiment of the present application, the query image encoder assists the second image encoder in encoding through the trained first image encoder, thereby improving the second image encoder's ability to express images and saving a large amount of computing resources in the model training process. Through this clever transfer learning strategy, not only the training speed of the second image encoder is accelerated, but also the performance and efficiency of the query image encoder are optimized.

[0143] In some embodiments, "based on the normalized fourth image coding feature, performing feature transformation on the third image coding feature to obtain the second image coding feature" can be achieved by the following steps: for each element in the normalized fourth image coding feature, when the element is less than a set threshold, determining the position of the element in the normalized fourth image coding feature; updating the element at the corresponding position in the third image coding feature, and determining the updated third image coding feature as the second image coding feature.

[0144] As an example, when the fourth image coding feature is processed based on a normalized exponential function (softmax function) to obtain a normalized fourth image coding feature, each element in the normalized fourth image coding feature is a real number greater than 0 or equal to 0, and less than 1 or equal to 1. When the element is less than a set threshold, the position of the element in the normalized fourth image coding feature is determined, the element at the corresponding position in the third image coding feature is updated to 0, and the updated third image coding feature is determined as the second image coding feature, wherein the set threshold is used to screen high-value elements in the normalized fourth image coding feature and to identify low-value elements in the fourth image coding feature, and the value of the element refers to the weight of the element in the fourth image coding feature.

[0145] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the weight determination method provided in the embodiment of the present application, Figure 3A The "fusion of the second image coding feature and the text coding feature to obtain the first fusion feature" in step 103 shown can be implemented through the following steps 1031 to 1033, which are described in detail below.

[0146] In step 1031, attention processing is performed on the second image coding feature to obtain a sixth attention feature.

[0147] As an example, based on the attention processing formula (1.7), the second image coding feature is cross-attention processed to obtain the sixth attention feature. The formula (1.7) is as follows:

[0148]

[0149] in, is the i-th row data of the sixth attention feature, O 8 ∈R N*256 , Qr is a learnable parameter, The i-th row of data for the second image encoding feature, query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, Qr i is the i-th row data of Qr, Qr j is the j-th row data of Qr, Sim is the distance metric function, and the distance metric function formula can be found in formula (1.3) shown above.

[0150] In step 1032, the sixth attention feature is linearly transformed to obtain a transformed sixth attention feature.

[0151] As an example, based on the linear layer, the sixth attention feature is dimensionally transformed to obtain the transformed sixth attention feature, and the feature dimension of the transformed sixth attention feature is N*512.

[0152] In step 1033, the transformed sixth attention feature and the text encoding feature are combined to obtain a first fusion feature.

[0153] As an example, based on the vertical stacking method, the transformed sixth attention feature and the text encoding feature are combined to obtain the first fusion feature, and the feature dimension of the first fusion feature is (N+K)*512.

[0154] In an embodiment of the present application, the second encoding model achieves further alignment between the second image encoding features and the text encoding features, so that after the second image encoding features and the prompt word text encoding features are fused, the expression of the query request for the object's weight is more accurate, laying the foundation for the subsequent output of the object's weight information through a large language model.

[0155] In step 104, based on the first fusion feature, weight prediction processing is performed on the object to obtain the weight of the object.

[0156] As an example, a large language model is called based on the first fusion feature to perform weight prediction on the object to obtain the weight of the object, wherein the embodiments of the present application are not limited to the model structure of the large language model. For example, the large language model can be a mixture model (Mixture Model) or a generative pre-trained transformer model (GPT, Generative Pre-trained Transformer).

[0157] In some embodiments, the first position coding feature is obtained by encoding through a first coding model, and the first image coding feature is obtained by encoding through a first image encoder, see Figure 3F , Figure 3F is a sixth flow chart of the weight determination method provided in the embodiment of the present application, Figure 3A Before step 101 shown, the training of the first image encoder and the first encoding model may be implemented by following steps 301 to 304, which are described in detail below.

[0158] In step 301, the positive image sample and the negative image sample are encoded respectively by using an initial first encoding model to obtain a second position encoding feature of the positive image sample and a third position encoding feature of the negative image sample.

[0159] As an example, the specific steps of encoding the positive sample and the negative sample of the image respectively by the initial first encoding model refer to the above step 101, which will not be repeated here. The second position encoding feature of the positive sample of the image can be represented as I P , I P ∈R N*256 , the third position encoding feature of the image negative sample can be characterized as I N , I N ∈R N*256 .

[0160] In step 302, the positive image sample and the negative image sample are respectively encoded by an initial first image encoder to obtain a fifth image encoding feature of the positive image sample and a sixth image encoding feature of the negative image sample.

[0161] As an example, the specific steps of encoding the positive image sample and the negative image sample by the initial first image encoder are as shown in step 1021 to step 1024 above, which will not be repeated here. The fifth image coding feature of the positive image sample can be represented as C P , C P ∈R N*256 , the sixth image encoding feature of the image negative sample can be represented as C N , C N ∈R N*256 , R is an arbitrary real number, and N is the number of image blocks.

[0162] In step 303, a first similarity between the second position coding feature and the fifth image coding feature is determined, and a second similarity between the third position coding feature and the sixth image coding feature is determined.

[0163] As an example, based on the formula (1.3) shown above, the first similarity sim(I) between the second position coding feature and the fifth image coding feature is determined. P ,C P ), and a second similarity sim(I between the third position coding feature and the sixth image coding feature N ,C N ).

[0164] In step 304, based on the first similarity and the second similarity, back propagation is performed on the initial first coding model and the initial first image encoder to obtain the first coding model and the first image encoder.

[0165] As an example, based on the first similarity and the second similarity, a first loss function (1.8) is constructed, and a first loss value of the first loss function is determined. Formula (1.8) is as follows:

[0166]

[0167] Among them, L con1 is the first loss value, P represents the positive sample of the image, N P is the number of positive samples of the image, N represents the negative samples of the image, and N N is the number of negative samples of the image, I P,i Represents the second position encoding feature of the positive sample of the i-th image, C P,i Represents the fifth image encoding feature of the i-th positive sample, I N,j Represents the third position encoding feature of the jth negative sample, C N,j Represents the sixth image encoding feature of the j-th negative sample.

[0168] By minimizing the first loss value, back-propagation is performed on the initial first encoding model and the initial first image encoder to obtain the first encoding model and the first image encoder.

[0169] The embodiment of the present application achieves a strong alignment capability between the first image encoder and the first encoding model by jointly training the first image encoder and the first encoding model.

[0170] In some embodiments, the first fusion feature is obtained by a second encoding model, see Figure 3G , Figure 3G is a seventh flow chart of the weight determination method provided in the embodiment of the present application, Figure 3A Before step 101 shown, the training of the second coding model can be implemented by following steps 401 to 403, which are described in detail below.

[0171] In step 401, the image coding features of the first image sample including the object sample are subjected to attention processing through the initial second coding model to obtain the seventh attention feature of the first image sample, and the seventh attention feature is subjected to linear transformation processing to obtain the transformed seventh attention feature.

[0172] As an example, the first image sample is a cutout image including an object sample, and the image coding feature of the first image sample is obtained by querying the image encoder and processing the first image sample. The image coding feature of the first image sample including the object sample is subjected to attention processing through the initial second coding model to obtain the seventh attention feature of the first image sample, and the seventh attention feature is subjected to linear transformation processing. The specific steps for obtaining the transformed seventh attention feature refer to steps 1031 and 1032 above, which will not be repeated here. The seventh attention feature is represented by I E , I E ∈R N*512 .

[0173] In step 402, text encoding is performed on the annotated text corresponding to the first image sample to obtain text encoding features of the annotated text.

[0174] As an example, based on the self-attention layer, the annotated text corresponding to the first image sample is subjected to attention processing to obtain the attention feature of the annotated text, and the attention feature of the annotated text is subjected to linear transformation processing to obtain the text encoding feature of the annotated text, wherein the text encoding feature of the annotated text is represented as I t , I t ∈R N*512 , the annotation text refers to the weight information text of the object, for example, the annotation text can be "The weight of this chicken is 3kg.".

[0175] In step 403, based on the similarity between the transformed seventh attention feature and the text encoding feature of the annotated text, the initial second encoding model is back-propagated to obtain the second encoding model.

[0176] Among them, the first image sample includes a positive sample and a negative sample, the transformed seventh attention feature includes the transformed seventh attention feature of the positive sample and the transformed seventh attention feature of the negative sample, and the text encoding feature of the annotated text includes the text encoding feature of the positive sample and the text encoding feature of the negative sample.

[0177] As an example, based on the formula (1.3) shown above, the third similarity Sim(I) between the transformed seventh attention feature of the positive sample and the text encoding feature of the positive sample is determined. E,P ,I t,P ), and determining the fourth similarity Sim(I) between the transformed seventh attention feature of the negative sample and the text encoding feature of the negative sample E,N ,I t,N ), based on the third similarity and the fourth similarity, construct a second loss function (1.9), and determine the second loss value of the second loss function. Formula (1.9) is as follows:

[0178]

[0179] Among them, L con2 is the second loss value, P represents the positive sample, N P is the number of positive samples, N represents negative samples, and N N is the number of negative samples, I E,P,i Represents the transformed seventh attention feature of the i-th positive sample, I t,P,i Represents the text encoding feature of the i-th positive sample, I E,N,j Represents the transformed seventh attention feature of the jth negative sample, I t,N,j represents the text encoding feature of the jth negative sample, Sim is the distance metric function, and the distance metric function formula is shown in formula (1.3) above.

[0180] By minimizing the second loss value, the initial second encoding model is back-propagated to obtain the second encoding model.

[0181] Next, an exemplary application of the embodiment of the present application in an actual chicken weight estimation application scenario will be described.

[0182] With the continuous advancement of artificial intelligence technology, its application in the field of animal farming has gradually deepened, especially in the key link of animal weighing. By automatically identifying individual animals, performing real-time monitoring and precise weighing, the intelligence and accuracy of farming management have been greatly improved. However, the related technology has a large demand for cameras and sensors, which may bring certain challenges to the initial investment and maintenance costs of farms.

[0183] In order to solve the above problems, an embodiment of the present application proposes a weight determination method, which only requires a single camera to complete the animal weight estimation work.

[0184] See also Figure 4 , Figure 4: is a first schematic diagram of the weight determination method provided in an embodiment of the present application. The weight determination method provided in an embodiment of the present application only includes three parts: a chicken identification part, a general segmentation part and a weight prediction part. In the chicken identification part, a chicken farm image (equivalent to the first image above) collected by on-site monitoring of the chicken farm is received, and the chicken farm image is processed by the counting model of the target detection model to output the bounding box (bbox) of the chicken (equivalent to the object above), that is, the upper left corner coordinates and the lower right corner coordinates, and the upper left corner coordinates and the lower right corner coordinates are converted into the center coordinates of the bounding box (equivalent to the position information above), which are recorded as the chicken target; in the general segmentation part, the input includes a chicken target and a chicken farm image, the chicken farm image is input into an image encoder (equivalent to the first image encoder above), and a chicken farm image coding feature (ImageToken, equivalent to the first image coding feature above) is obtained, and the chicken target information is input into a conditional encoder (equivalent to the first coding model above) to obtain a conditional coding feature (conditionTok en, equivalent to the position coding feature above), input ImageToken and conditionToken into the mask decoder (equivalent to the mask generator above), and obtain the mask image of the chicken (equivalent to the mask feature above); in the weight prediction part, firstly, the mask image of the chicken is subjected to the corresponding cutout processing on the chicken farm picture to obtain the mask cutout (equivalent to the second image above), and the mask cutout is input into the query image encoder to obtain the chicken image coding feature (equivalent to the second image coding feature above). At the same time, this application implements the design of prompt words similar to "How much does this chicken weigh?" (equivalent to the prompt text above) as the input of the prompt word encoder, and obtains the coding feature of the prompt word (equivalent to the text coding feature of the prompt text above), and then inputs the coding feature of the prompt word and the coding feature of the chicken image into the coding fusion (equivalent to the second coding model above), and obtains the fusion feature (equivalent to the first fusion feature above), and finally inputs the fusion feature into the large language model to obtain the final weight information. It is worth noting that the above three parts are independent of each other and have their own unique training strategies.

[0185] The following will respectively Figure 4 The general segmentation part and weight prediction part in this paper are introduced.

[0186] 1. General segmentation

[0187] 1) Image Encoder

[0188] See also Figure 5 , Figure 5 is a second principle diagram of the weight determination method provided in an embodiment of the present application, the input of the image encoder is an image I, I∈R H*W*C, where R is a real number, H is the height of the input image, W is the width of the input image, and C is the number of channels of the input image. The input image is processed as follows to obtain the output chicken farm image encoding features (image tokens) of the image encoder. Figure 5 The structure of the image encoder is introduced.

[0189] Step 501, image splitting (patch splitting): split the input image into patches, and divide the input image into N fixed-size patches, where N = n 2 , the size of each patch is p*p, At this time, the dimension of each patch is represented by D = p 2 *C, after patch splitting, the size of the input image I′ after patch splitting is I′∈R N*D .

[0190] Step 502: Input activation-attention layer processing: The input image I after patch splitting ′ Input activation-attention layer, based on the formula (1.2) shown above, the input image I after patch splitting ′ Processing is performed to obtain the output O of the activation-attention layer, O∈R N*D , where the query value Q i =I′ i *W Q , key value K j =I′ j *W K , value V i =I′ i *W V , W Q is the weight corresponding to the query value, W K is the weight corresponding to the key value, W V is the weight corresponding to the value, W Q , W K , W V ∈R D*f , f is a hyperparameter, I′ i is the i-th row data of the input image I′ after patch splitting, O i is the i-th row data of the output O of the activation-attention layer, O i ∈R 1*D , Sim is the distance measurement function, and the distance measurement function formula can be found in formula (1.3) shown above.

[0191] Step 503, deep convolution layer processing: perform edge padding processing (Padding operation) on the input image I to obtain an edge-filled image I″∈RF*F*D , F is a hyperparameter, and a deep convolutional layer is used to extract features from the edge-filled image I″. The size of the convolution kernel is F*F, and the output of the deep convolutional layer is represented as O′=DWCon(I), O′∈R 1*D .

[0192] Step 504: Concatenate and combine: The output O of the activation-attention layer and the output O of the deep convolution layer are concatenated by matrix stacking. ′ After splicing, we get T o , T o ∈R (N+1)*D .

[0193] Step 505, linear regression: In order to unify the output dimension of the conditional encoder below, a linear regression method is used to obtain the T obtained after splicing and combination. o Perform linear transformation to obtain the chicken farm image encoding feature imageToken, imageToken∈R N*256 .

[0194] 2) Conditional Encoder

[0195] See also Figure 6 , Figure 6 The third principle diagram of the weight determination method provided in the embodiment of the present application, the input of the conditional encoder is the coordinates (x, y) of the center point of the chicken output by the chicken identification part, recorded as the condition (condition, equivalent to the position information above), and the following will be combined with Figure 6 The structure of the conditional encoder is introduced.

[0196] Step 601, point prompt processing: perform point prompt processing on the chicken center point coordinate (x, y) information, and change the single point (x, y) to (x, y, l), l∈{-1 / 1}, 1 indicates that the coordinate point is on the chicken target, and -1 indicates that the point is not on the chicken target. Among them, the coordinate of the chicken center point must be on the chicken target, so it is recorded as (x, y, 1).

[0197] Step 602, point prompt embedding: Based on the prompt word embedding layer, the result (x, y, 1) after point prompt processing is used as input, and embedding processing is performed using a learnable nonlinear function (equivalent to the mapping processing above). The formula of the learnable nonlinear function is shown in the above formula (1.1), where, b∈R 1*256 , the output of the word embedding layer C p is a hyperparameter and g is an activation function.

[0198] Step 603: Flatten the stack: Use the activation function to ensure that the subsequent output is a fixed number of tokens, and perform edge padding or masking on the output of the prompt word embedding layer to obtain Where N is a fixed value.

[0199] Step 604, linear transformation: Use linear regression to transform V into conditional encoding feature conditionToken, conditionToken∈R N*256 .

[0200] 3) Mask decoder

[0201] See also Figure 7 , Figure 7 : is the fourth principle diagram of the weight determination method provided in the embodiment of the present application. The input of the mask decoder is the chicken farm image encoding feature imageToken, denoted as I, and the conditional encoding feature conditionToken, denoted as C, where I∈R N*256 , C∈R N*256 , the following will combine Figure 7 The structure of the mask decoder is introduced.

[0202] Step 701, attention layer 1 processing: I and C are input into attention layer 1 for processing. The calculation formula of attention layer 1 is shown in formula (1.4) above, where the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, I i is the i-th row data of I, C i is the i-th row data of C, is the output O of attention layer 1 1 The i-th row data, O 1 ∈R N*512 , Sim is the distance metric function.

[0203] Step 702, attention layer 2 processing: I and C are input into attention layer 2 for processing. The calculation formula of attention layer 2 is shown in formula (1.5) above, where the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, I i is the i-th row data of I, C i is the i-th row data of C, is the output O of attention layer 2 2 The i-th row data, O 2 ∈R N*256 , Sim is the distance metric function.

[0204] Step 703, linear layer 1 processing: The output of attention layer 2 is subjected to feature extraction processing through the learnable linear layer 1 to obtain O 3 , O 3 ∈R N*512 .

[0205] Step 704, attention layer 3 processing: convert O 1 and O 3 The input is processed in the attention layer 3. The calculation formula of the attention layer 3 is shown in formula (1.6) shown above, where the query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, O 1 The i-th row of data, O 3 The i-th row of data, is the output O of attention layer 3 4 The i-th row data, O 4 ∈R N*1024 , Sim is the distance metric function.

[0206] Step 705, linear layer 2 processing: The output of the attention layer 3 is subjected to feature extraction processing by the learnable linear layer 2 to obtain O 5 , O 5 ∈R N*2048 .

[0207] Step 706, upsampling layer processing: In order to obtain the final segmentation result, the linear layer 2 output O is processed by the upsampling layer. 5 Perform upsampling to obtain the output result mask image O of the mask decoder 6 , O 6 ∈R H*W*1 .

[0208] The training process of the general segmentation part includes two stages, which will be introduced below.

[0209] Stage 1: Training the image encoder and conditional encoder. As mentioned above, the output of the image encoder is I, I∈R N*256 , the output of the conditional encoder is C, C∈R N*256 By comparing the learning strategies, the conditional encoder and the image encoder are trained. The training formula is shown in formula (1.8) above, where P represents the positive sample and N P is the number of positive samples, N represents negative samples, and N N is the number of negative samples, I P,i represents the output result of the image encoder corresponding to the i-th positive sample, C P,i Represents the output result of the conditional encoder corresponding to the i-th positive sample, I N,j represents the output result of the image encoder corresponding to the jth negative sample, C N,j represents the output result of the conditional encoder corresponding to the jth negative sample, L con1 The embodiment of the present application uses a large amount of target detection data set to train this part in the training data, thereby achieving a strong alignment capability between the image encoder and the conditional encoder.

[0210] Stage 2: Training the mask decoder. The mask decoder is trained using the cross entropy loss function, which will not be described here.

[0211] 2. Weight prediction part

[0212] 1) Query the image encoder

[0213] See also Figure 8 , Figure 8 This is the fifth principle diagram of the weight determination method provided in the embodiment of the present application. Figure 8 The structure of the query image encoder is introduced.

[0214] Step 801, performing cutout according to the mask image: receiving the mask image output by the general segmentation part, and obtaining a cutout image according to the mask image.

[0215] Step 802, image encoder processing: input the cutout image into the image encoder (equivalent to the first image encoder above) for processing to obtain 1 , O 1 ∈R N*256 .

[0216] Step 803, retrained image encoder processing: input the cutout image into the retrained image encoder (equivalent to the second image encoder above) for processing, and obtain 2 , O 2 ∈R N*256, based on the normalized exponential function (softmax function), 2 Process and filter out high-value parts based on set thresholds.

[0217] Step 804: Query: Based on 2 The high-value part is screened and O is taken out 1 The values ​​corresponding to the medium and high value parts are set to 0, and the rest are set to 0 to obtain the chicken image coding feature O 3 , O 3 ∈R N*256 .

[0218] 2) Encoding Fusion

[0219] See also Fig. 9 , Fig. 9 6 is a sixth principle diagram of the weight determination method provided in the embodiment of the present application, wherein the input of the encoding fusion device is the chicken image encoding feature O 3 , the prompt word text encoding feature P extracted by the prompt word encoder 1 (promptToken) and a learnable query value Qr, and the output of the encoding fusion is used as the input of the large language model (LLM), where O 3 ∈R N*256 , P 1 ∈R K*512 , Qr 1 ∈R N*256 , K is a hyperparameter, and the following will combine Figure 6 The structure of the coding fusion device is introduced.

[0220] Step 901, cross attention layer processing: Qr and O 3 The input is processed in the cross attention layer. The calculation formula (2.1) of the cross attention layer is as follows:

[0221]

[0222] The query value Key-value value is the weight corresponding to the query value, is the weight corresponding to the key value, is the weight corresponding to the value, Qr j is the j-th row data of Qr, O 3 The i-th row of data, is the output O of the cross attention layer 4 The i-th row data, O 4 ∈R N*256 , Sim is the distance metric function.

[0223] Step 902: Linear transformation: Perform linear transformation on the output of the cross attention layer to obtain the image embedding feature I after linear transformation. E (imageEmbedding, equivalent to the mapping feature above), I E ∈R N*512 .

[0224] Step 903: Fusion: P 1 and O 4 Stacked together, we get the fusion feature L1, L1∈R (K+N)*512 , input L1 into the large language model.

[0225] See also Fig.10 , Fig.10 This is the seventh principle diagram of the weight determination method provided in the embodiment of the present application. Fig.10 The training process of the encoder fusion is introduced.

[0226] In order to enable the encoder fusion device to have the function of estimating weight, the embodiment of the present application defines the training data set as {image: chicken image cropped according to mask information, prompt word text: "How much does this chicken weigh?", result text: "This chicken weighs 1Kg"}.

[0227] In the training phase of the encoder fusion, the image is input into the encoder fusion to obtain the image embedding feature I E , the detailed steps here can be referred to step 901 to step 903 above, which will not be repeated here; the result text (equivalent to the annotated text above) is processed by self-attention to obtain the encoded features after self-attention processing, and the encoded features after self-attention processing are linearly transformed through the linear layer to obtain the encoded features of the result text I t , I t ∈R N*512 .

[0228] Based on contrastive learning, the loss value of the loss function is minimized, the parameters of the cross-attention layer are updated, the learnable query, the parameters of the linear layer 1, the parameters of the cross-attention layer, the parameters of the linear layer 2, and the loss function is shown in formula (1.9) shown above, where P represents the positive sample, N P is the number of positive samples, N represents negative samples, and N N is the number of negative samples, I E,P,i Represents the image embedding feature corresponding to the i-th positive sample, I t,P,i Represents the encoding features of the result text corresponding to the i-th positive sample, I E,N,j Represents the image embedding feature corresponding to the jth negative sample, I t,N,j Represents the encoding features of the result text corresponding to the jth negative sample, Lcon2 is the loss value of contrastive learning.

[0229] The beneficial effects of the embodiments of the present application are mainly reflected in the following aspects:

[0230] 1) In the related art, in the traditional convolutional neural network (CNN) architecture, the image encoder often ignores the consideration of the condition position information, which limits its effectiveness in the condition matching task and makes the matching ability relatively weak. However, the image encoder in the embodiment of the present application adopts a patch splitting and patch arrangement strategy. This structural design has significant advantages in revealing the image position information. It can not only capture and map the local features of the image with higher accuracy, but also establish a closer and more intuitive corresponding matching relationship with the condition, which greatly improves the matching performance and effect of the encoder. In addition, in view of the problem that the image encoder based on the multi-modality of pictures and texts generally adopts the Transformer architecture with a large number of parameters and slow calculation, the embodiment of the present application provides a lightweight general image encoder. While maintaining high performance, this encoder greatly reduces the number of parameters and computational complexity, providing a more flexible and efficient solution for image coding tasks.

[0231] 2) The conditional encoder based on multi-modality of images and texts mostly adopts the Transformer architecture, which has large amount of computation and slow calculation. Therefore, this application implements a lightweight universal conditional encoder.

[0232] 3) The instance segmentation model in the related art needs to rely on a large amount of professional data training (such as chickens). When the amount of data is insufficient, it is difficult to achieve good results. For this reason, the present application implements a mask decoder of ImageToken and conditionToken, and generates a mask image. The key point information is used as the condition (the output (x, y) of the chicken recognition part) to specify the region-level prior information for the segmentation condition, which can further control the area to be segmented. Therefore, compared with the traditional segmentation model in the related art, the mask image generated by the mask decoder provided in the embodiment of the present application improves the accuracy of mask image generation.

[0233] 4) In the weight prediction part, retraining a new image encoder also requires a large amount of sample data. Therefore, a query-based image encoder is designed, which utilizes the trained image encoder in general segmentation to assist in encoding the image input of the weight prediction module, thereby improving the weight prediction module's ability to express images.

[0234] 5) In order to achieve further alignment between image coding features and weight information, an encoding fusion device is designed so that after the fusion of image coding features and prompt word text coding features, the query request for the weight of the chicken is more accurate, laying the foundation for the subsequent output of chicken weight information through a large language model.

[0235] In summary, the embodiments of the present application greatly reduce the cost of manual annotation and the hardware cost of camera sensors through powerful general visual capabilities (image encoder, conditional encoder, query image encoder, etc.) and general text generation capabilities based on large prediction models.

[0236] The following is a description of an exemplary structure of the weight determination device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the weight determination device 555 of the memory 550 may include:

[0237] The position coding module 5551 is used to encode the position information of the object in the first image to obtain a first position coding feature of the object.

[0238] The image coding module 5552 is used to determine a first image coding feature of the first image.

[0239] The mask generating module 5553 is used to extract a second image including the object from the first image based on the first image coding feature and the first position coding feature.

[0240] The image encoding module 5552 is further used to determine a second image encoding feature of the second image.

[0241] The coding fusion module 5554 is used to determine the text coding feature of the prompt text, fuse the second image coding feature and the text coding feature to obtain a first fusion feature, wherein the prompt text is used to indicate the weight of the generated object.

[0242] The weight determination module 5555 is used to perform weight prediction processing on the object based on the first fusion feature to obtain the weight of the object.

[0243] In some embodiments, the image encoding module 5552 is also used to encode the first image to obtain a first encoding feature of the first image; perform convolution processing on the first image to obtain a convolution feature of the first image; perform fusion processing on the first encoding feature and the convolution feature to obtain a second fusion feature; perform linear transformation processing on the second fusion feature to obtain the first image encoding feature.

[0244] In some embodiments, the image encoding module 5552 is also used to split the first image to obtain multiple image blocks of the first image; perform image encoding on each image block to obtain a second encoding feature of each image block; and splice multiple second encoding features to obtain a first encoding feature of the first image.

[0245] In some embodiments, the mask generation module 5553 is also used to perform a first attention processing on the first image coding feature and the first position coding feature to obtain a first attention feature; perform a second attention processing on the first image coding feature and the first position coding feature to obtain a second attention feature; perform a third attention processing on the first attention feature and the second attention feature to obtain a third attention feature; and determine the second image from the first image based on the third attention feature.

[0246] In some embodiments, the mask generation module 5553 is further configured to perform cross-attention processing on the first position coding feature based on the first image coding feature to obtain a fourth attention feature;

[0247] Perform feature extraction processing on the fourth attention feature to obtain the second attention feature.

[0248] In some embodiments, the mask generation module 5553 is also used to perform feature extraction processing on the third attention feature to obtain a fifth attention feature; perform upsampling processing on the fifth attention feature to obtain a mask feature; and based on the mask feature, perform object extraction processing on the object included in the first image to obtain the second image including the object.

[0249] In some embodiments, the first image coding feature is obtained by encoding the first image with a first image encoder, and the image coding module 5552 is further used to encode the second image with the first image encoder to obtain a third image coding feature, and encode the second image with a second image encoder to obtain a fourth image coding feature; normalize the fourth image coding feature to obtain the normalized fourth image coding feature; and perform feature transformation on the third image coding feature based on the normalized fourth image coding feature to obtain the second image coding feature.

[0250] In some embodiments, the image coding module 5552 is further used to determine, for each element in the normalized fourth image coding feature, a position of the element in the normalized fourth image coding feature when the element is less than a set threshold; update the element corresponding to the position in the third image coding feature, and determine the updated third image coding feature as the second image coding feature.

[0251] In some embodiments, the coding fusion module 5554 is also used to perform attention processing on the second image coding feature to obtain a sixth attention feature; perform linear transformation processing on the sixth attention feature to obtain the transformed sixth attention feature; and combine the transformed sixth attention feature and the text coding feature to obtain the first fusion feature.

[0252] In some embodiments, the position encoding module 5551 is also used to map the position information to obtain the mapping features of the position information; perform sequence alignment on the mapping features of the position information to obtain the aligned mapping features; perform linear transformation on the aligned mapping features to obtain the first position encoding features.

[0253] In some embodiments, the first fusion feature is obtained through a second coding model, and the device also includes a training module, which is used to perform attention processing on the image coding features of a first image sample including an object sample through an initial second coding model to obtain a seventh attention feature of the first image sample, and perform linear transformation processing on the seventh attention feature to obtain the transformed seventh attention feature; perform text encoding processing on the annotated text corresponding to the first image sample to obtain the text coding feature of the annotated text; based on the similarity between the transformed seventh attention feature and the text coding feature of the annotated text, back-propagate the initial second coding model to obtain the second coding model.

[0254] In some embodiments, the first position coding feature is obtained by encoding through a first coding model, and the first image coding feature is obtained by encoding through the first image encoder. The device also includes a training module, and the training module is used to encode the image positive sample and the image negative sample respectively through the initial first coding model to obtain the second position coding feature of the image positive sample and the third position coding feature of the image negative sample; encode the image positive sample and the image negative sample respectively through the initial first image encoder to obtain the fifth image coding feature of the image positive sample and the sixth image coding feature of the image negative sample; determine the first similarity between the second position coding feature and the fifth image coding feature, and determine the second similarity between the third position coding feature and the sixth image coding feature; based on the first similarity and the second similarity, back-propagate the initial first coding model and the initial first image encoder to obtain the first coding model and the first image encoder.

[0255] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the weight determination method described in the embodiment of the present application.

[0256] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the weight determination method provided in the present application embodiment, for example, Figures 3A to 3G The weight determination method is shown.

[0257] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0258] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0259] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0260] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0261] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for determining weight, characterized in that: The method comprises: Encoding position information of the object in the first image to obtain a first position coding feature of the object; determining a first image coding feature of the first image, and extracting a second image including the object from the first image based on the first image coding feature and the first position coding feature; Determine a second image coding feature of the second image, determine a text coding feature of a prompt text, and fuse the second image coding feature and the text coding feature to obtain a first fused feature, wherein the prompt text is used to indicate the weight of the generated object; Based on the first fusion feature, weight prediction processing is performed on the object to obtain the weight of the object.

2. The method according to claim 1, characterized in that The determining of a first image coding feature of the first image comprises: Encoding the first image to obtain a first encoding feature of the first image; Performing convolution processing on the first image to obtain convolution features of the first image; Fusing the first coding feature and the convolution feature to obtain a second fused feature; Performing linear transformation processing on the second fusion feature to obtain the first image coding feature.

3. The method according to claim 2, characterized in that The step of encoding the first image to obtain a first encoding feature of the first image includes: Performing splitting processing on the first image to obtain a plurality of image blocks of the first image; Performing image encoding on each image block to obtain a second encoding feature of each image block; A plurality of the second coding features are spliced ​​together to obtain a first coding feature of the first image.

4. The method according to claim 1, characterized in that: The step of extracting a second image including the object from the first image based on the first image coding feature and the first position coding feature comprises: Performing a first attention processing on the first image coding feature and the first position coding feature to obtain a first attention feature; Performing a second attention processing on the first image coding feature and the first position coding feature to obtain a second attention feature; Performing third attention processing on the first attention feature and the second attention feature to obtain a third attention feature; Based on the third attention feature, the second image is determined from the first image.

5. The method according to claim 4, characterized in that The performing second attention processing on the first image coding feature and the first position coding feature to obtain a second attention feature includes: Based on the first image coding feature, performing cross attention processing on the first position coding feature to obtain a fourth attention feature; Perform feature extraction processing on the fourth attention feature to obtain the second attention feature.

6. The method according to claim 4, characterized in that The step of determining the second image from the first image based on the third attention feature comprises: performing feature extraction processing on the third attention feature to obtain a fifth attention feature; Performing upsampling processing on the fifth attention feature to obtain a mask feature; Based on the mask feature, object extraction processing is performed on the object included in the first image to obtain the second image including the object.

7. The method according to claim 1, characterized in that The first image coding feature is obtained by encoding the first image by a first image encoder, and determining the second image coding feature of the second image includes: Encoding the second image by using the first image encoder to obtain a third image coding feature, and encoding the second image by using the second image encoder to obtain a fourth image coding feature; Normalizing the fourth image coding feature to obtain the normalized fourth image coding feature; Based on the normalized fourth image coding feature, feature transformation is performed on the third image coding feature to obtain the second image coding feature.

8. The method according to claim 7, characterized in that The step of performing feature transformation on the third image coding feature based on the normalized fourth image coding feature to obtain the second image coding feature includes: For each element in the normalized fourth image coding feature, if the element is less than a set threshold, determine the position of the element in the normalized fourth image coding feature; An element corresponding to the position in the third image coding feature is updated, and the updated third image coding feature is determined as the second image coding feature.

9. The method according to claim 1, characterized in that: The fusing the second image coding feature and the text coding feature to obtain a first fused feature includes: performing attention processing on the second image encoding feature to obtain a sixth attention feature; Performing a linear transformation process on the sixth attention feature to obtain the transformed sixth attention feature; The transformed sixth attention feature and the text encoding feature are combined to obtain the first fusion feature.

10. The method according to claim 1, characterized in that The step of encoding the position information of the object in the first image to obtain a first position coding feature of the object includes: Performing mapping processing on the position information to obtain mapping features of the position information; Performing sequence alignment on the mapping features of the position information to obtain the aligned mapping features; The aligned mapping features are linearly transformed to obtain the first position coding features.

11. A weight determination device, characterized in that: The device comprises: A position coding module, used for coding the position information of the object in the first image to obtain a first position coding feature of the object; an image coding module, configured to determine a first image coding feature of the first image; a mask generation module, configured to extract a second image including the object from the first image based on the first image coding feature and the first position coding feature; The image coding module is further used to determine a second image coding feature of the second image; A coding fusion module, used for determining a text coding feature of a prompt text, fusing the second image coding feature and the text coding feature to obtain a first fusion feature, wherein the prompt text is used to indicate the weight of the generated object; A weight determination module is used to perform weight prediction processing on the object based on the first fusion feature to obtain the weight of the object.

12. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the weight determination method according to any one of claims 1 to 10 when executing the computer executable instructions or computer program stored in the memory.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the weight determination method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the weight determination method according to any one of claims 1 to 10 is implemented.

Images