A multimodal visual recognition method for chicken slaughtering and segmentation and bone removal

Abstract

The present invention relates to the technical field of computer-aided raw chicken slaughter data supervision and processing. The present invention provides a multimodal visual recognition method for chicken slaughter segmentation and bone removal. The present invention constructs a digital model by acquiring equipment information, simulates and optimizes lighting conditions, establishes an industrial camera matrix to collect high-quality images, uses image recognition technology to accurately identify chicken parts, extracts and fuses features of images that do not meet the recognition standards, continuously trains and optimizes the recognition model, and finally applies it to the monitoring of the actual slaughtering process. This method significantly improves the accuracy and efficiency of chicken slaughtering, and at the same time solves the impact of unstable lighting on image recognition during the segmentation of raw chicken bones, bringing technological innovation to the chicken slaughtering industry.

Description

Technical Field

[0001] The invention relates to the technical field of computer-aided raw chicken slaughtering data supervision processing, and in particular to a chicken slaughtering segmentation and bone removal multimodal visual recognition method. Background Art

[0002] In the process of chicken slaughtering and cutting, bone removal is a key link. The traditional manual operation method is not only inefficient, but also has hygiene and safety issues. Therefore, the introduction of multimodal visual recognition technology and the precise cutting and deboning of chicken through automated equipment have become an important way to improve production efficiency and product quality.

[0003] The application of multimodal visual recognition technology in chicken slaughtering, first of all, the collected chicken images are preprocessed, including image enhancement, denoising and other steps to improve the image quality and provide an accurate data basis for subsequent processing. The RGB image is converted to Lab color space, and the image is segmented using the Kmeans clustering algorithm. This method can effectively separate the chicken from the bone area. Deep learning models such as SSD (Single Shot MultiBox Detector) are introduced to further identify and locate the segmented images. The model is trained to identify the bone area and achieve accurate bone removal. In addition to image information, other sensor data (such as X-ray images, mechanical sensors, etc.) can also be combined for multimodal information fusion to improve the accuracy and robustness of recognition. For example, X-ray images can help identify the bone structure inside the chicken, thereby more accurately guiding the bone removal operation.

[0004] During the process of chicken slaughtering, segmenting and bone removal, the lighting conditions at the slaughter site are unstable and the light brightness varies greatly, which directly affects the quality of image acquisition. Too dark or too bright light will cause the loss of image details, which in turn affects the accuracy of image recognition. Due to inaccurate image recognition data, the trained model has deviations and cannot accurately reflect the actual situation, resulting in incorrect segmentation of chicken during the segmentation process, causing too much chicken to be removed and resulting in reduced product quality. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a multimodal visual recognition method for removing bones from slaughtered chicken, which solves the problem that too dark or too bright light will cause the loss of image details, thereby affecting the accuracy of image recognition. Due to inaccurate image recognition data, the trained model has deviations and cannot accurately reflect the actual situation, resulting in the problem of mis-segmentation of chicken during the segmentation process.

[0006] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0007] The present invention provides a multimodal visual recognition method for removing bones from slaughtered and segmented chicken, comprising:

[0008] S101, obtaining chicken slaughtering, splitting and bone removal equipment information, the chicken slaughtering, splitting and bone removal equipment information including deboning machine information, splitting machine information and conveyor belt information, retrieving the position information of the deboning machine information, splitting machine information and conveyor belt information, and obtaining the equipment position information;

[0009] S102, collecting building information of the area where the device is located according to the device location information, the building information including the location of the building wall and the location of the building window, substituting the device location information and the building information of the area where the device is located into a preset digital twin model to obtain a digital twin image of the area where the device is located;

[0010] S103, performing data analysis on the digital twin image of the area where the equipment is located through a natural environment simulation model to obtain a natural environment simulation result, the natural environment simulation result including simulation information of sunlight shining on a deboning machine, a splitting machine, and a conveyor belt, setting a lighting device in the area where the equipment is located according to the natural environment simulation result, and configuring a preset lighting rule for the lighting device;

[0011] S104, establishing an industrial camera matrix according to the equipment position information, the industrial camera matrix including a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, configuring a set of industrial camera matrices for each of the deboning machine, the splitting machine, and the conveyor belt, collecting image data of the equipment in operation through the industrial camera matrix, substituting the image data collected by the industrial camera matrix in the equipment in operation into a preset image recognition model, and identifying the position and shape of the chicken legs, chicken wings, and chicken breasts;

[0012] S105, detect the position and shape of chicken legs, chicken wings and chicken breasts. If the detection result does not meet the expected standard, extract features from the image data collected by the industrial camera matrix in the running state, fuse the extracted features, use the fused features to train the image recognition model, optimize the parameters of the trained image recognition model to obtain a trained multimodal visual recognition model, and deploy the trained multimodal visual recognition model to monitor the removal of chicken bones during the slaughtering process of raw chickens.

[0013] Furthermore, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S101, obtains chicken slaughtering, segmenting and bone removal equipment information, the chicken slaughtering, segmenting and bone removal equipment information includes bone removal machine information, segmentation machine information and conveyor belt information, retrieves the position information of the bone removal machine information, the segmentation machine information and the conveyor belt information, and obtains the equipment position information, including:

[0014] The server receives a request from a backend control terminal or a user terminal for obtaining information about a chicken slaughtering, cutting and bone removal device;

[0015] The server queries the database for basic information of the equipment according to the request for obtaining information of the chicken slaughtering, segmenting and bone removal equipment. The basic information of the equipment includes information of the bone remover, segmenting machine and conveyor belt. The information of the bone remover, segmenting machine and conveyor belt includes equipment model, equipment status and equipment production capacity.

[0016] After obtaining the basic information of the device, a communication connection is established with the device, the built-in GPS module of the device and the device location information preset in the factory are called up, and the location information of the GPS module is compared with the device location information preset in the factory. If the positions in the comparison results are consistent, the device location information is generated. If the comparison results are inconsistent, the device location error information is generated.

[0017] Furthermore, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention, S102, collects building information of the area where the equipment location is located according to the equipment location information, the building information includes the building wall location and the building window location, substitutes the equipment location information and the building information of the area where the equipment location is located into a preset digital twin model, and obtains a digital twin image of the area where the equipment location is located, including:

[0018] The server parses the device location information to obtain the device coordinate information and device height information;

[0019] Based on the coordinate information of the device, the server collects the building information of the area where the coordinate information of the device is located, and the building information includes the location of the building wall and the location of the window;

[0020] Preprocess the coordinate information of the device, the height information of the device, and the building information, and convert them into a coordinate system consistent with the preset digital twin model, and obtain the coordinate information of the device, the height information of the device, and the building information after the coordinate conversion;

[0021] The server substitutes the coordinate information of the device after the coordinate conversion, the height information of the device, and the building information into the preset digital twin model. Based on the substituted information, the digital twin model generates a digital twin image of the area where the device is located;

[0022] The digital twin image of the area where the equipment is located is used to show the location of the equipment and the surrounding environment within the building, including the location of walls and windows.

[0023] Furthermore, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention also includes:

[0024] The server stores the digital twin image of the area where the device is located in the database, and encrypts the digital twin image of the area where the device is located through a preset privacy computing model to obtain encrypted image data after privacy computing;

[0025] Establish administrator rights and corresponding data processing servers for administrator rights. When the administrator completes the decryption of the encrypted image data after privacy calculation through the preset rights;

[0026] The server then decrypts the encrypted image data after the privacy calculation and transmits the image data obtained to the data processing server corresponding to the administrator's authority for subsequent data processing.

[0027] Furthermore, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S103, performs data analysis on the digital twin image of the area where the equipment is located through a natural environment simulation model to obtain a natural environment simulation result, the natural environment simulation result includes simulation information of sunlight irradiating on a deboning machine, a segmenting machine and a conveyor belt, and sets a lighting device in the area where the equipment is located according to the natural environment simulation result, and configures a preset lighting rule for the lighting device, including:

[0028] Determine the natural environment phenomena that the natural environment simulation model needs to simulate, and the natural environment phenomena that need to be simulated are sunlight, air humidity, and visual visibility information;

[0029] According to the equipment location information and building information, the geographical scope and time of the simulation are determined to obtain the simulation target, and the multi-source data of the simulation target is collected. The multi-source data of the simulation target includes the climate data in the factory and the data of sunlight shining on the equipment. The data is pre-processed to obtain the pre-processed multi-source data of the simulation target;

[0030] The preprocessed multi-source data of the simulation target are used to train the ecosystem dynamics model, the parameters and variables of the ecosystem dynamics model are determined, and the trained ecosystem dynamics model is obtained;

[0031] The trained ecosystem dynamics model is trained using historical data or experimental data to reflect the dynamic changes of the actual natural environment, and the trained ecosystem dynamics model is evaluated. The trained ecosystem dynamics model is optimized using an optimization algorithm to obtain a natural environment simulation model.

[0032] Furthermore, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention also includes:

[0033] The server integrates the digital twin model and the natural environment simulation model, and integrates the algorithms, databases and interfaces of the digital twin model and the natural environment simulation model, so that the digital twin model and the natural environment simulation model can communicate with each other and share data.

[0034] The digital twin model is used to provide the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory, and transmits the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory to the natural environment simulation model. The natural environment simulation model generates the lighting information, humidity information, and visual visibility information in the factory, and feeds the lighting information, humidity information, and visual visibility information in the factory back to the digital twin model;

[0035] The simulation results of the natural environment simulation model are updated in real time and fed back to the digital twin model. The accuracy of the model is verified by comparing it with the actual environmental data. If the data comparison results are deviated, the natural environment simulation model is adjusted and optimized.

[0036] Furthermore, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S104, establishes an industrial camera matrix according to the equipment position information, the industrial camera matrix includes a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, a deboning machine, a segmenting machine and a conveyor belt are each equipped with a set of industrial camera matrices, and image data of the equipment under the running state is collected by the industrial camera matrix, and the image data collected by the industrial camera matrix under the running state of the equipment is substituted into a preset image recognition model to identify the position and shape of chicken legs, chicken wings and chicken breasts, including:

[0037] When the device is running, the server will synchronously start the industrial camera matrix corresponding to the device, pre-process and store the image data collected by the industrial camera matrix, and substitute the image data collected by the industrial camera matrix in the running state of the device into the preset image recognition model;

[0038] The server needs to use a preset occlusion detection algorithm to detect whether there are occlusions in the image data when the device is running. The occlusion detection analysis results are obtained by comparing the differences in similar parts of the chicken after segmentation between consecutive frames of the image data when the device is running, and analyzing the brightness and texture features in the image.

[0039] Further, S105, the position and shape of the chicken legs, chicken wings and chicken breasts are detected. If the detection result does not meet the expected standard, feature extraction is performed on the image data collected by the industrial camera matrix in the running state, and the extracted features are fused, and the image recognition model is trained using the fused features. The trained image recognition model is parameter optimized to obtain a trained multimodal visual recognition model, and the trained multimodal visual recognition model is deployed to monitor the removal of chicken bones during the slaughtering process of raw chickens, including:

[0040] If the detection result does not meet the expected standard, query the corresponding occlusion detection analysis result in the detection result, if there is occlusion information in the occlusion detection analysis result;

[0041] The occluded area is located through the edge detection model, and the occluded area is repaired through the preset image repair model to obtain the image data of the occluded area after repair, and the image data of the occluded area after repair is replaced with the original data, and the replaced data is subjected to feature extraction and fusion.

[0042] The beneficial effects of the present invention are as follows:

[0043] Improve monitoring accuracy and efficiency: By establishing an industrial camera matrix and combining it with a preset image recognition model, the present invention can more comprehensively and accurately capture and record every detail of the chicken slaughter, segmentation and bone removal process. This not only improves the monitoring accuracy, but also increases the number of detailed images in the segmentation process due to multi-angle image acquisition, thereby improving data processing efficiency.

[0044] Optimize lighting configuration: By using the natural environment simulation model for data analysis, the lighting in the area where the equipment is located can be adjusted and optimized according to actual needs, ensuring the quality of image acquisition and further improving the clarity of monitoring.

[0045] Realize the generation and storage of digital twin images: The present invention can generate and store digital twin images of the area where the equipment is located. This not only helps in real-time monitoring, but can also be used for subsequent data analysis and processing, thus increasing the use value of the data.

[0046] Improved automation: By training and deploying a multimodal visual recognition model, the present invention can automatically monitor the chicken slaughtering, segmentation and bone removal during the raw chicken slaughtering process, reducing the need for manual intervention and improving the automation level.

[0047] In summary, the present invention significantly improves the monitoring accuracy, efficiency and automation of the chicken slaughtering process through technological innovation, and brings substantial improvements to the chicken slaughtering industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of a process method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail below in conjunction with the drawings.

[0051] In order to better understand the purpose of the present invention, the present invention is described in further detail below.

[0052] The present invention provides a multimodal visual recognition method for removing bones from slaughtered and segmented chicken, comprising:

[0053] In step S101, chicken slaughtering, splitting and bone removal equipment information is obtained, the chicken slaughtering, splitting and bone removal equipment information includes deboning machine information, splitting machine information and conveyor belt information, the position information of the deboning machine information, splitting machine information and conveyor belt information is retrieved, and the equipment position information is obtained;

[0054] First, step S101 can determine the precise location of the device, providing basic and necessary information for the subsequent establishment of a digital twin model, configuration of a camera matrix, and setting of lighting devices. Secondly, by obtaining device information, the server can monitor the operating status of the device, ensure that the device is working properly, and promptly detect potential problems, thereby improving production efficiency and product quality. In addition, obtaining accurate device information also provides a solid foundation for the subsequent fusion of multi-source data such as image data and illumination data, which helps to achieve multimodal visual recognition.

[0055] In terms of real-time monitoring, the acquisition of equipment information enables the server to obtain the chicken processing status of the deboning machine, splitting machine and conveyor belt in real time, and combine it with the precise location information of the equipment to achieve accurate monitoring of specific areas. At the same time, based on the equipment location information, the server can optimize the lighting conditions to ensure that the camera can collect high-quality image data under different lighting conditions. Finally, this equipment information also helps the server to provide timely feedback and adjustments. When it is found that the chicken splitting or deboning results do not meet expectations, the feedback mechanism can be triggered immediately to continuously optimize the recognition model to ensure the accuracy and robustness of the monitoring system.

[0056] In step S102, building information of the area where the device is located is collected according to the device location information, the building information includes the building wall position and the building window position, and the device location information and the building information of the area where the device is located are substituted into a preset digital twin model to obtain a digital twin image of the area where the device is located;

[0057] Step S102 collects building information in the area where the device is located, especially the wall positions and window positions, and substitutes it into the digital twin model to generate an accurate digital twin image of the area where the device is located.

[0058] By analyzing the device location information and collecting building information, the server can generate a digital twin image that reflects the actual layout and surrounding environment of the device in reality. This not only helps to accurately locate the device, but also displays detailed information about the environment around the device.

[0059] Digital twin images provide a basis for subsequent natural environment simulations, simulating the impact of different lighting conditions on equipment, optimizing lighting settings, and ensuring the quality of image acquisition. At the same time, digital twin models can update equipment status and environmental changes in real time, providing an intuitive monitoring interface that allows managers to quickly respond to abnormal situations.

[0060] Wall location information helps analyze the path of sunlight, determine areas with sufficient light and areas that may be blocked, and thus reasonably arrange lighting devices in the actual environment to improve image acquisition quality. At the same time, understanding the location of the wall can also optimize the spatial layout of the production workshop, improve monitoring efficiency and production safety.

[0061] Window position information is essential for making the best use of natural light and adjusting lighting fixtures. By adjusting lighting based on window position and orientation, you can reduce energy consumption for artificial lighting while maintaining the quality of image acquisition. In addition, by monitoring the location of windows and the weather conditions around them, you can predict changes in light intensity and adjust lighting settings in advance.

[0062] In step S103, the digital twin image of the area where the equipment is located is analyzed through the natural environment simulation model to obtain a natural environment simulation result, which includes simulation information of sunlight shining on the deboning machine, the splitting machine and the conveyor belt. According to the natural environment simulation result, a lighting device is set in the area where the equipment is located, and a preset lighting rule is configured for the lighting device;

[0063] The server uses the device location information and building information, combined with the digital twin model, to generate a digital twin image of the area where the device is located. Then, the image is analyzed using a natural environment simulation model to simulate natural environmental phenomena such as sunlight. This process collects and processes multi-source data to ensure data accuracy.

[0064] The natural environment simulation model then outputs simulated information about sunlight shining on key equipment (deboning machines, splitters, and conveyor belts). This information is real-time and dynamic, reflecting the lighting conditions under different time and environmental conditions. Based on these simulation results, the server sets up appropriate lighting devices in the area where the equipment is located and configures preset lighting rules.

[0065] Under any natural lighting conditions, the slaughter site can be ensured to obtain stable and appropriate lighting, avoiding the loss of image details and improving the quality of image acquisition. Stable lighting conditions provide a high-quality data foundation for subsequent image recognition, enabling the image recognition model to more accurately identify the position and shape of chicken. Optimizing lighting conditions can reduce image recognition errors caused by unstable lighting, thereby reducing the risk of mis-segmentation during the chicken segmentation process and improving product quality and production efficiency.

[0066] In step S104, an industrial camera matrix is ​​established according to the equipment position information, the industrial camera matrix includes a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, and a set of industrial camera matrices are configured for each of the deboning machine, the splitting machine, and the conveyor belt. The image data of the equipment in the running state is collected through the industrial camera matrix, and the image data collected through the industrial camera matrix in the running state of the equipment is substituted into a preset image recognition model to identify the position and shape of the chicken legs, chicken wings, and chicken breasts;

[0067] First, the server will establish an industrial camera matrix based on the equipment location information. This matrix consists of industrial cameras at multiple angles, including cameras from angles 1 to 6. This setting can ensure comprehensive monitoring of the deboning machine, cutter, and chicken on the conveyor belt from different perspectives, effectively avoiding monitoring blind spots and ensuring data integrity and accuracy.

[0068] Next, the server will synchronously start the industrial camera matrix corresponding to the equipment to collect and process the image data of the equipment in real time. This real-time feedback mechanism can detect and handle abnormal situations in time to ensure the smooth progress of the slaughtering process.

[0069] The collected image data will be substituted into the preset image recognition model to identify the position and shape of chicken legs, wings and breasts. This recognition capability is based on algorithms such as deep learning, ensuring high accuracy and robustness, and maintaining stable recognition performance even in complex slaughtering environments.

[0070] In addition, the server also uses an occlusion detection algorithm to detect whether there are obstructions, and locates and repairs the occluded area through an edge detection model and an image repair model, thereby ensuring that the data received by the recognition model is complete and accurate.

[0071] In step S105, the position and shape of the chicken legs, chicken wings and chicken breast are detected. If the detection result does not meet the expected standard, feature extraction is performed on the image data collected by the industrial camera matrix in the running state, and the extracted features are fused. The fused features are used to train the image recognition model, and the parameters of the trained image recognition model are optimized to obtain a trained multimodal visual recognition model. The trained multimodal visual recognition model is deployed to monitor the chicken slaughtering, segmentation and bone removal during the raw chicken slaughtering process.

[0072] First, the system will start the configured industrial camera matrix synchronously to collect comprehensive image data of the equipment in operation. After pre-processing, the data is substituted into the preset image recognition model for automatic recognition to locate and identify the position and shape of chicken legs, chicken wings and chicken breasts.

[0073] If the detection result does not meet the expected standards, the server will optimize the recognition model. This includes feature extraction, extracting key information such as image brightness and texture, feature fusion, fusing these features into a more comprehensive feature set to improve recognition accuracy and robustness, model training, using these features to train the image recognition model to improve its performance; parameter optimization, further improving its generalization ability and recognition accuracy by adjusting model parameters, and finally, deploying the trained and optimized multimodal visual recognition model to the system.

[0074] In this way, the system can obtain high-quality image data in real time and achieve accurate recognition through advanced image processing and machine learning technology. This automated visual recognition not only reduces the need for manual intervention and improves production efficiency, but also accurate recognition and real-time monitoring help maintain high product quality and consistency, thereby reducing labor costs and avoiding errors caused by human factors.

[0075] Specifically, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S101, obtains chicken slaughtering, segmenting and bone removal equipment information, the chicken slaughtering, segmenting and bone removal equipment information includes bone removal machine information, segmentation machine information and conveyor belt information, retrieves the position information of the bone removal machine information, the segmentation machine information and the conveyor belt information, and obtains the equipment position information, including:

[0076] The server receives a request from a backend control terminal or a user terminal for obtaining information about a chicken slaughtering, cutting and bone removal device;

[0077] The server queries the database for basic information of the equipment according to the request for obtaining information of the chicken slaughtering, segmenting and bone removal equipment. The basic information of the equipment includes information of the bone remover, segmenting machine and conveyor belt. The information of the bone remover, segmenting machine and conveyor belt includes equipment model, equipment status and equipment production capacity.

[0078] After obtaining the basic information of the device, a communication connection is established with the device, the built-in GPS module of the device and the device location information preset in the factory are called up, and the location information of the GPS module is compared with the device location information preset in the factory. If the positions in the comparison results are consistent, the device location information is generated. If the comparison results are inconsistent, the device location error information is generated.

[0079] Specifically, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention, S102, collects building information of the area where the equipment is located according to the equipment location information, the building information includes the building wall position and the building window position, substitutes the equipment location information and the building information of the area where the equipment is located into a preset digital twin model, and obtains a digital twin image of the area where the equipment is located, including:

[0080] The server parses the device location information to obtain the device coordinate information and device height information;

[0081] Based on the coordinate information of the device, the server collects the building information of the area where the coordinate information of the device is located, and the building information includes the location of the building wall and the location of the window;

[0082] Preprocess the coordinate information of the device, the height information of the device, and the building information, and convert them into a coordinate system consistent with the preset digital twin model, and obtain the coordinate information of the device, the height information of the device, and the building information after the coordinate conversion;

[0083] The server substitutes the coordinate information of the device after the coordinate conversion, the height information of the device, and the building information into the preset digital twin model. Based on the substituted information, the digital twin model generates a digital twin image of the area where the device is located;

[0084] The digital twin image of the area where the equipment is located is used to show the location of the equipment and the surrounding environment within the building, including the location of walls and windows.

[0085] Specifically, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention also includes:

[0086] The server stores the digital twin image of the area where the device is located in the database, and encrypts the digital twin image of the area where the device is located through a preset privacy computing model to obtain encrypted image data after privacy computing;

[0087] Establish administrator rights and corresponding data processing servers for administrator rights. When the administrator completes the decryption of the encrypted image data after privacy calculation through the preset rights;

[0088] The server then decrypts the encrypted image data after the privacy calculation and transmits the image data obtained to the data processing server corresponding to the administrator's authority for subsequent data processing.

[0089] Specifically, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S103, performs data analysis on the digital twin image of the area where the equipment is located through a natural environment simulation model to obtain a natural environment simulation result, the natural environment simulation result includes simulation information of sunlight irradiating on a deboning machine, a segmenting machine and a conveyor belt, and sets a lighting device in the area where the equipment is located according to the natural environment simulation result, and configures a preset lighting rule for the lighting device, including:

[0090] Determine the natural environment phenomena that the natural environment simulation model needs to simulate, and the natural environment phenomena that need to be simulated are sunlight, air humidity, and visual visibility information;

[0091] According to the equipment location information and building information, the geographical scope and time of the simulation are determined to obtain the simulation target, and the multi-source data of the simulation target is collected. The multi-source data of the simulation target includes the climate data in the factory and the data of sunlight shining on the equipment. The data is pre-processed to obtain the pre-processed multi-source data of the simulation target;

[0092] The preprocessed multi-source data of the simulation target are used to train the ecosystem dynamics model, the parameters and variables of the ecosystem dynamics model are determined, and the trained ecosystem dynamics model is obtained;

[0093] The trained ecosystem dynamics model is trained using historical data or experimental data to reflect the dynamic changes of the actual natural environment, and the trained ecosystem dynamics model is evaluated. The trained ecosystem dynamics model is optimized using an optimization algorithm to obtain a natural environment simulation model.

[0094] Specifically, the chicken slaughtering and segmenting bone removal multimodal visual recognition method provided by the present invention also includes:

[0095] The server integrates the digital twin model and the natural environment simulation model, and integrates the algorithms, databases and interfaces of the digital twin model and the natural environment simulation model, so that the digital twin model and the natural environment simulation model can communicate with each other and share data.

[0096] The digital twin model is used to provide the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory, and transmits the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory to the natural environment simulation model. The natural environment simulation model generates the lighting information, humidity information, and visual visibility information in the factory, and feeds the lighting information, humidity information, and visual visibility information in the factory back to the digital twin model;

[0097] The simulation results of the natural environment simulation model are updated in real time and fed back to the digital twin model. The accuracy of the model is verified by comparing it with the actual environmental data. If the data comparison results are deviated, the natural environment simulation model is adjusted and optimized.

[0098] Specifically, the chicken slaughtering, segmenting and bone removal multimodal visual recognition method provided by the present invention, S104, establishes an industrial camera matrix according to the equipment position information, the industrial camera matrix includes a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, a deboning machine, a segmenting machine and a conveyor belt are each equipped with a set of industrial camera matrices, and image data of the equipment under the running state is collected by the industrial camera matrix, and the image data collected by the industrial camera matrix under the running state of the equipment is substituted into a preset image recognition model to identify the position and shape of chicken legs, chicken wings and chicken breasts, including:

[0099] When the device is running, the server will synchronously start the industrial camera matrix corresponding to the device, pre-process and store the image data collected by the industrial camera matrix, and substitute the image data collected by the industrial camera matrix in the running state of the device into the preset image recognition model;

[0100] The server needs to use a preset occlusion detection algorithm to detect whether there are occlusions in the image data when the device is running. The occlusion detection analysis results are obtained by comparing the differences in similar parts of the chicken after segmentation between consecutive frames of the image data when the device is running, and analyzing the brightness and texture features in the image.

[0101] Specifically, S105 detects the position and shape of chicken legs, chicken wings and chicken breasts. If the detection result does not meet the expected standard, extract features from the image data collected by the industrial camera matrix in the running state, fuse the extracted features, use the fused features to train the image recognition model, optimize the parameters of the trained image recognition model, obtain a trained multimodal visual recognition model, and deploy the trained multimodal visual recognition model to monitor the removal of chicken bones during the slaughtering process of raw chickens, including:

[0102] If the detection result does not meet the expected standard, query the corresponding occlusion detection analysis result in the detection result, if there is occlusion information in the occlusion detection analysis result;

[0103] The occluded area is located through the edge detection model, and the occluded area is repaired through the preset image repair model to obtain the image data of the occluded area after repair, and the image data of the occluded area after repair is replaced with the original data, and the replaced data is subjected to feature extraction and fusion.

[0104] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments when they are aware of the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0105] The specific implementation of the present invention in practical scenarios is as follows:

[0106] Obtaining device information. In actual scenarios, the server receives a request from the backend control terminal or the user terminal to obtain information about the chicken slaughtering, segmenting and bone removal equipment. After the server queries the basic information of the equipment, it establishes a communication connection with the equipment and obtains the equipment location information.

[0107] Collect building information and generate digital twin images. Based on the coordinate information of the device, the server collects the building information of the area where the device is located and generates a digital twin image of the area where the device is located, showing the specific location of the device and the surrounding environment.

[0108] Natural environment simulation and lighting configuration, the server uses the natural environment simulation model to perform data analysis on the digital twin image, and configures the lighting device according to the simulation results to ensure high-quality image data under different lighting conditions.

[0109] Establish an industrial camera matrix and collect image data. Configure the industrial camera matrix according to the equipment location information, collect image data when the equipment is in operation, and use the occlusion detection algorithm to analyze the image data.

[0110] Image recognition and model training: Substitute image data into the image recognition model to identify the position and shape of chicken legs, chicken wings and chicken breasts. If the detection results do not meet expectations, use the fused features to train a new recognition model and deploy it to the system for real-time monitoring.

[0111] Data encryption and permission management: The server encrypts and stores digital twin images, establishes administrator permissions and data processing servers, and ensures data security and privacy.

[0112] Model optimization and verification, the server integrates digital twin models and natural environment simulation models to achieve communication and data sharing between models, and verifies the accuracy of the model by comparing it with actual environmental data, and makes adjustments and optimizations when necessary.

[0113] Through the above steps, the present invention improves the monitoring accuracy and efficiency of the chicken slaughtering process in actual scenarios, bringing improvements to the chicken slaughtering industry.

[0114] The present invention solves the technical problem that the image details are lost due to too dark or too bright light, which in turn affects the accuracy of image recognition and causes the wrong segmentation of chicken meat by the following methods:

[0115] First, by obtaining the equipment location information and generating digital twin images, the system can accurately simulate the layout of the equipment and its surrounding environment. Secondly, the natural environment simulation model is used to optimize the lighting to ensure that the equipment area can obtain appropriate lighting under different lighting conditions, avoiding the loss of image details due to lighting problems. At the same time, a multi-angle industrial camera matrix is ​​constructed to collect image data in all directions to capture more image details. In terms of image recognition, recognition is performed through a preset image recognition model, and the model is iteratively trained and optimized multiple times to improve the accuracy of recognition. In addition, occlusion detection and image restoration technology are used to ensure the integrity of image data. These measures together improve the accuracy of image recognition, thereby reducing the problem of mis-segmentation in the chicken segmentation process.

[0116] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations, and the above-described embodiments of the present invention do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A multimodal visual recognition method for removing bones from chicken slaughtering and segmentation, characterized in that: include: S101, obtaining chicken slaughtering, splitting and bone removal equipment information, the chicken slaughtering, splitting and bone removal equipment information including deboning machine information, splitting machine information and conveyor belt information, retrieving the position information of the deboning machine information, splitting machine information and conveyor belt information, and obtaining the equipment position information; S102, collecting building information of the area where the device is located according to the device location information, the building information including the location of the building wall and the location of the building window, substituting the device location information and the building information of the area where the device is located into a preset digital twin model to obtain a digital twin image of the area where the device is located; S103, performing data analysis on the digital twin image of the area where the equipment is located through a natural environment simulation model to obtain a natural environment simulation result, the natural environment simulation result including simulation information of sunlight shining on a deboning machine, a splitting machine, and a conveyor belt, setting a lighting device in the area where the equipment is located according to the natural environment simulation result, and configuring a preset lighting rule for the lighting device; S104, establishing an industrial camera matrix according to the equipment position information, the industrial camera matrix including a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, configuring a set of industrial camera matrices for each of the deboning machine, the splitting machine, and the conveyor belt, collecting image data of the equipment in operation through the industrial camera matrix, substituting the image data collected by the industrial camera matrix in the equipment in operation into a preset image recognition model, and identifying the position and shape of the chicken legs, chicken wings, and chicken breasts; S105, detecting the position and shape of the chicken legs, chicken wings and chicken breasts. If the detection result does not meet the expected standard, extracting features from the image data collected by the industrial camera matrix in the running state, fusing the extracted features, training an image recognition model using the fused features, optimizing the parameters of the trained image recognition model, obtaining a trained multimodal visual recognition model, and deploying the trained multimodal visual recognition model to monitor the removal of chicken bones during the slaughtering process of raw chickens; S103, performing data analysis on the digital twin image of the area where the equipment is located through a natural environment simulation model to obtain a natural environment simulation result, which includes simulation information of sunlight shining on a deboning machine, a splitting machine, and a conveyor belt. According to the natural environment simulation result, a lighting device is set in the area where the equipment is located, and preset lighting rules are configured for the lighting device, including: Determine the natural environment phenomena that the natural environment simulation model needs to simulate, and the natural environment phenomena that need to be simulated are sunlight, air humidity, and visual visibility information; According to the equipment location information and building information, the geographical scope and time of the simulation are determined to obtain the simulation target, and the multi-source data of the simulation target is collected. The multi-source data of the simulation target includes the climate data in the factory and the data of sunlight shining on the equipment. The data is pre-processed to obtain the pre-processed multi-source data of the simulation target; The preprocessed multi-source data of the simulation target are used to train the ecosystem dynamics model, the parameters and variables of the ecosystem dynamics model are determined, and the trained ecosystem dynamics model is obtained; Using historical data or experimental data to train the trained ecosystem dynamics model to reflect the dynamic changes of the actual natural environment, and evaluating the trained ecosystem dynamics model, and using an optimization algorithm to optimize the trained ecosystem dynamics model to obtain a natural environment simulation model; Also includes: The server integrates the digital twin model and the natural environment simulation model, and integrates the algorithms, databases and interfaces of the digital twin model and the natural environment simulation model, so that the digital twin model and the natural environment simulation model can communicate with each other and share data. The digital twin model is used to provide the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory, and transmits the location of the equipment in the factory, the location of the factory windows, and the height of the equipment in the factory to the natural environment simulation model. The natural environment simulation model generates the lighting information, humidity information, and visual visibility information in the factory, and feeds the lighting information, humidity information, and visual visibility information in the factory back to the digital twin model; The simulation results of the natural environment simulation model are updated in real time and fed back to the digital twin model. The accuracy of the model is verified by comparing it with the actual environmental data. If the data comparison results are deviated, the natural environment simulation model is adjusted and optimized.

2. The multimodal visual recognition method for removing bones from slaughtered and segmented chicken according to claim 1, characterized in that: S101, obtaining chicken slaughtering, splitting and bone removal equipment information, the chicken slaughtering, splitting and bone removal equipment information including deboning machine information, splitting machine information and conveyor belt information, retrieving the position information of the deboning machine information, splitting machine information and conveyor belt information, and obtaining the equipment position information, including: The server receives a request from a backend control terminal or a user terminal for obtaining information about a chicken slaughtering, cutting and bone removal device; The server queries the database for basic information of the equipment according to the request for obtaining information of the chicken slaughtering, segmenting and bone removal equipment. The basic information of the equipment includes information of the bone remover, segmenting machine and conveyor belt. The information of the bone remover, segmenting machine and conveyor belt includes equipment model, equipment status and equipment production capacity. After obtaining the basic information of the device, a communication connection is established with the device, the built-in GPS module of the device and the device location information preset in the factory are called up, and the location information of the GPS module is compared with the device location information preset in the factory. If the positions in the comparison results are consistent, the device location information is generated. If the comparison results are inconsistent, the device location error information is generated.

3. The multimodal visual recognition method for removing bones from slaughtered and segmented chicken according to claim 1, characterized in that: S102, collecting building information of the area where the device is located according to the device location information, the building information includes the building wall location and the building window location, substituting the device location information and the building information of the area where the device is located into a preset digital twin model, and obtaining a digital twin image of the area where the device is located, including: The server parses the device location information to obtain the device coordinate information and device height information; Based on the coordinate information of the device, the server collects the building information of the area where the coordinate information of the device is located, and the building information includes the location of the building wall and the location of the window; Preprocess the coordinate information of the device, the height information of the device, and the building information, and convert them into a coordinate system consistent with the preset digital twin model, and obtain the coordinate information of the device, the height information of the device, and the building information after the coordinate conversion; The server substitutes the coordinate information of the device after the coordinate conversion, the height information of the device, and the building information into the preset digital twin model. Based on the substituted information, the digital twin model generates a digital twin image of the area where the device is located; The digital twin image of the area where the equipment is located is used to show the location of the equipment and the surrounding environment within the building, including the location of walls and windows.

4. The multimodal visual recognition method for removing bones from slaughtered and segmented chicken according to claim 3, characterized in that: Also includes: The server stores the digital twin image of the area where the device is located in the database, and encrypts the digital twin image of the area where the device is located through a preset privacy computing model to obtain encrypted image data after privacy computing; Establish administrator rights and corresponding data processing servers for administrator rights. When the administrator completes the decryption of the encrypted image data after privacy calculation through the preset rights; The server then decrypts the encrypted image data after the privacy calculation and transmits the image data obtained to the data processing server corresponding to the administrator's authority for subsequent data processing.

5. The multimodal visual recognition method for removing bones from slaughtered and segmented chicken according to claim 1, characterized in that: S104, establishing an industrial camera matrix according to the equipment position information, the industrial camera matrix including a first angle industrial camera, a second angle industrial camera, a third angle industrial camera, a fourth angle industrial camera, a fifth angle industrial camera, and a sixth angle industrial camera, configuring a set of industrial camera matrices for each of the deboning machine, the splitting machine, and the conveyor belt, collecting image data of the equipment in operation through the industrial camera matrix, substituting the image data collected by the industrial camera matrix in the equipment in operation into a preset image recognition model, and identifying the position and shape of the chicken legs, chicken wings, and chicken breasts, including: When the device is running, the server will synchronously start the industrial camera matrix corresponding to the device, pre-process and store the image data collected by the industrial camera matrix, and substitute the image data collected by the industrial camera matrix in the running state of the device into the preset image recognition model; The server needs to use a preset occlusion detection algorithm to detect whether there are occlusions in the image data when the device is running. The occlusion detection analysis results are obtained by comparing the differences in similar parts of the chicken after segmentation between consecutive frames of the image data when the device is running, and analyzing the brightness and texture features in the image.

6. The multimodal visual recognition method for removing bones from slaughtered and segmented chicken according to claim 1, characterized in that: S105, detecting the position and shape of the chicken legs, chicken wings and chicken breasts. If the detection result does not meet the expected standard, extracting features from the image data collected by the industrial camera matrix in the running state, fusing the extracted features, training the image recognition model with the fused features, optimizing the parameters of the trained image recognition model, obtaining a trained multimodal visual recognition model, and deploying the trained multimodal visual recognition model to monitor the removal of chicken bones during the slaughtering of raw chickens, including: If the detection result does not meet the expected standard, query the corresponding occlusion detection analysis result in the detection result, if there is occlusion information in the occlusion detection analysis result; The occluded area is located through the edge detection model, and the occluded area is repaired through the preset image repair model to obtain the image data of the repaired occluded area, and the repaired image data of the occluded area is replaced with the original data, and the replaced data is subjected to feature extraction and fusion.

Images