Interference filtering method for key recognition, intelligent musical instrument, equipment and medium

Abstract

The invention relates to the field of playing digitization, in particular to a key recognition method, an intelligent musical instrument, equipment and a medium. The method comprises the steps of obtaining a first view image; executing a first filtering operation on the first view image to suppress the image noise of the first form, and obtaining a second view image; preset threshold processing is carried out on the second visual field image to obtain a third visual field image, the third visual field image comprises a plurality of target pixel sets, and each target pixel set is composed of a plurality of continuous pixel points with pixel values lower than a preset threshold; executing a second filtering operation on a target pixel set in the third visual field image based on a preset structural element to suppress image noise in a second form, and obtaining a fifth visual field image; and based on a second preset model, identifying a plurality of first-class keys from the fifth view image. Therefore, high-precision interference elimination is realized while key details are reserved, and the recognition precision and robustness of keys in a complex scene are effectively improved.

Description

[0001] Divisional application This application is a divisional application of Chinese invention patent application filed on May 8, 2025 [Application No.: 2025105856261] [Title: A Piano Key Recognition Method, Intelligent Musical Instrument, Device and Medium]. Technical Field

[0002] This application relates to the field of digital performance, and more particularly to an interference filtering method for piano key recognition, a smart musical instrument, a device, and a medium. Background Technology

[0003] With the development of technology, the digital expression of music and art is becoming increasingly diverse, and it has broad application prospects in education, entertainment, virtual reality, and other fields. Among them, piano key recognition technology is a core technology in music teaching, performance assistance, and the development of intelligent musical instruments. Its core objective is to detect the position and status of piano keys through sensors or vision systems.

[0004] For example, patent application CN111695499A discloses a piano key recognition method, device, electronic device, and storage medium. The method includes: acquiring a first keyboard image of a target musical instrument, wherein the keyboard of the target musical instrument is provided with black keys and white keys; inputting the first keyboard image into a preset image recognition model to determine the outline of each black key in the first keyboard image; and determining the piano key range corresponding to each octave and the outline of each white key in each octave in the first keyboard image based on the outline of the black keys and the spacing between adjacent black keys.

[0005] For example, Chinese invention patent application CN113723264A discloses a method for intelligently identifying playing errors to assist piano teaching, which includes: acquiring a 2D image of the piano being played from above the piano keyboard, containing the complete piano keyboard; performing target detection on the 2D image through a piano keyboard detection network to detect the piano keyboard area represented by the relative position coordinates of the 2D image, and obtaining the piano keyboard position coordinates in the original coordinates of the 2D image through transformation.

[0006] However, in practical applications, environmental noise such as natural light, stage lighting, and moving objects can affect the quality of the piano key images captured by the vision system, leading to a decrease in the accuracy of piano key recognition and insufficient robustness in complex scenarios. Summary of the Invention

[0007] The main objective of this application is to provide an interference filtering method for piano key recognition, an intelligent musical instrument, a device, and a medium. To solve the aforementioned technical problems, this application specifically adopts the following technical solution: A first aspect of this application is to provide a piano key recognition method, the method comprising: S301, Obtain a first field-of-view image, wherein the first field-of-view image includes several images corresponding to a class of piano keys; S302, perform a first filtering operation on the pixels in the first field of view image based on the pixel values ​​in the first field of view image to suppress the image noise of the first form and obtain a second field of view image; S303, the second field of view image is processed by a preset threshold to obtain a third field of view image. The third field of view image includes a number of target pixel sets, which are composed of a number of consecutive pixels with pixel values ​​lower than the preset threshold. S304, perform a second filtering operation on the target pixel set in the third field of view image based on a preset structural element to suppress the second type of image noise and obtain a fifth field of view image; S305, based on the second preset model, identify several piano keys of the first type from the fifth field of view image.

[0008] A second aspect of this application is to provide an intelligent musical instrument, the intelligent musical instrument comprising: An image acquisition module is used to acquire a first field-of-view image, which includes several images corresponding to a class of piano keys. Memory, used to store computer programs; A processor is configured to execute the computer program and, in executing the computer program, implement the steps of the piano key recognition method provided in any embodiment of this application.

[0009] A third aspect of this application is to provide a computer device, the device comprising: Memory, used to store computer programs; A processor is configured to execute the computer program and, in executing the computer program, implement the steps of the piano key recognition method provided in any embodiment of this application.

[0010] A fourth aspect of this application is that a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the piano key recognition method provided in any embodiment of this application.

[0011] Beneficial effects: This application proposes a piano key recognition method, an intelligent musical instrument, a device, and a medium. Specifically, it provides a phased, progressive interference filtering mechanism under strict temporal conditions. The filtering intensity of each stage is strictly limited, and only specific interference information is processed. In the early stage, the original image data is highly respected for fine-tuning. In the later stage, adaptive correction is performed based on the reliable image data after processing and camera parameters. During the process, some interference information is tolerated and passed on to the next stage before being gradually eliminated. This avoids the loss of details caused by one-time high-sensitivity filtering. Thus, high-precision interference removal is achieved while preserving piano key details, effectively improving the recognition accuracy and robustness of piano keys in complex scenarios.

[0012] The first stage only performs local smoothing on discrete, isolated noise points (such as sensor noise or dust reflections). The multi-directional smoothing strategy based on pixel distribution characteristics can maintain the overall structure of the image and avoid excessive noise reduction that could lead to loss of detail. Edge blurring that may be caused by the smoothing strategy is cleared and restored during thresholding, ensuring the geometric integrity of the keys and thus providing a reliable basis for parameter calibration in the second stage.

[0013] In the second stage, pixel aggregation with conservative parameter settings is used to fill in the micro-fractures inside the black keys caused by the shooting angle (such as micro-fractures inside the black keys caused by wood texture or uneven lighting), reconstruct the complete closed area, and then implement fine boundary trimming to simultaneously eliminate structural noise (such as burrs on the edges of piano keys, keyboard seams and linear interference formed by reflections) and background interference. This achieves simultaneous optimization of repairing effective features and stripping ineffective features, avoiding multiple broken edges generated by unclosed contours in the subsequent extraction process, which would lead to chaotic contour hierarchy relationships.

[0014] In the third stage, the key outline is adaptively corrected based on perspective deformation. By dividing the keys near and far from the optical axis into regions and correcting dynamic parameters, the calibrated key outline restores the deformation error caused by the shooting angle, so as to accurately locate the keys at each position in the image. Even when the instrument model and shooting angle change, it can still maintain a high recognition rate and has a stronger generalization ability.

[0015] The fourth stage involves physical verification of the keyboard layout parameters. To avoid premature pruning that could lead to the accidental deletion of effective contours due to local deformation, physical verification is performed at the end of the process. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale. Obviously, the drawings described below are some embodiments of this application; for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0017] Figure 1 This is a schematic flowchart of a piano key recognition method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a first field-of-view image provided in an embodiment of this application; Figure 3 This is a schematic diagram of a second field-of-view image provided in an embodiment of this application; Figure 4 This is a schematic diagram of a third-view image provided in an embodiment of this application; Figure 5 This is a schematic diagram of a fifth field-of-view image provided in an embodiment of this application; Figure 6 This is a schematic diagram of a type of piano key contour extraction provided in an embodiment of this application; Figure 7 This is a schematic diagram of another type of piano key wheel extraction provided in the embodiments of this application; Figure 8 This is a schematic diagram of a type of piano key with adjusted outline provided in an embodiment of this application; Figure 9 This is a schematic flowchart of a piano key recognition method based on key seams provided in an embodiment of this application; Figure 10 This is a schematic diagram of a first partial image cropping provided in an embodiment of this application; Figure 11 This is a schematic diagram of a key recognition method based on key seams provided in an embodiment of this application; Figure 12 This is a schematic diagram of a piano key recognition result provided in an embodiment of this application; Figure 13 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] In this document, suffixes such as “module,” “part,” or “unit” used to denote elements are used only for illustrative purposes and have no specific meaning in themselves. Therefore, “module,” “part,” or “unit” may be used interchangeably.

[0020] In this document, the terms "upper," "lower," "inner," "outer," "front," "rear," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0021] In this document, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," and "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0022] In this document, "and / or" includes any and all combinations of one or more of the listed related items.

[0023] In this article, "multiple" means two or more, that is, it includes two, three, four, five, etc.

[0024] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0025] In practical applications, the environment in which piano keys are located is highly variable, and the image acquisition module can be set up in various positions. Current piano key recognition technology has low environmental adaptability, making it difficult to maintain high reliability in real-world applications and limiting its widespread use. On one hand, variations in natural light or stage lighting introduce additional noise, causing overexposure, glare, or color distortion in the images captured by the image acquisition module, reducing the accuracy of feature extraction. Furthermore, complex environmental backgrounds (such as moving objects or cluttered backgrounds) cause the target to blend into the background, increasing the difficulty of recognition and significantly reducing accuracy. On the other hand, different settings of the image acquisition module lead to changes in viewing angle, causing the piano keys to appear in different states in the image, further increasing the difficulty of recognition.

[0026] Based on this, this application proposes a piano key recognition method, specifically providing a staged, progressive interference filtering mechanism under strict temporal conditions. The filtering intensity of each stage is strictly limited, and only specific interference information is processed. In the early stage, the original image data is highly respected for fine-tuning, and in the later stage, adaptive correction is performed based on the reliable image data and camera parameters after processing. During the process, some noise or blurry information is tolerated to be transmitted to subsequent stages, gradually eliminating various interference information in the image and avoiding the loss of details caused by one-time high-sensitivity filtering. Thus, high-precision interference removal is achieved while preserving piano key details, effectively improving the recognition accuracy and robustness of piano keys in complex scenes.

[0027] Furthermore, this application proposes a key recognition method based on key seams. It first identifies a class of keys with prominent features, quickly pinpointing the local area of ​​a second class of keys to be identified. Based on deformation compensation using dynamic structural elements, it normalizes the prominent key seams within the local area, and then uses these seams to infer the regional contours of the keys. Thus, by employing a multi-level region-of-interest convergence strategy of "global image - local image - key seams," combined with the coupling mechanism of dynamic structural elements and perspective deformation parameters, it achieves both efficient utilization of computational resources and optimization of recognition accuracy, while effectively improving the recognition's anti-interference capability.

[0028] In this article, the musical instrument has keys arranged in a fixed order, and each key has a fixed pitch register. These keys can be combined to form a keyboard. The specific musical instrument can be a piano, organ, accordion, or electronic keyboard, etc., without limitation.

[0029] In this article, the image acquisition module is a device that converts optical signals in the physical world into digital images, such as cameras and camcorders. Its core components may include photoelectric converters (such as CCD or CMOS sensors), lenses, signal processing units, and storage modules. The model and specific components of the image acquisition module can be selected according to actual needs and are not limited here.

[0030] For example, the installation location of the image acquisition module can be determined based on specific application scenarios (such as ambient light) and technical parameters (such as resolution, installation height, angle, and image integrity) to ensure device performance and compliance. The specific installation location is not limited here. For instance, the image acquisition module can be perpendicular to the piano key plane, located directly above the keyboard, such as inside the piano lid, on the top of the piano stand, or on a hanging bracket. Alternatively, the image acquisition module can be installed at a certain angle to the piano key plane, on the front or upper side of the piano. Yet another example is that the image acquisition module can be independently positioned at the user's viewing angle.

[0031] In this paper, perspective is a fundamental principle in optical imaging, describing the "nearer objects appear larger and farther objects appear smaller" deformation of three-dimensional objects in two-dimensional images due to distance differences. Correspondingly, perspective deformation parameters reflect the deformation characteristics of piano keys caused by the camera's perspective, such as the shrinkage ratio and shrinkage direction. The perspective is lower near the image acquisition module than at the edge, and the corresponding perspective deformation parameters change accordingly.

[0032] It should be understood that the image acquisition module has preset internal and external parameters, namely preset camera parameters, including focal length, installation height, pitch angle, horizontal deflection angle, sensor size, lens distortion coefficient, etc., which can be used to calibrate the optical axis position. Based on these preset camera parameters and the object's position in the image (such as the coordinates of a type I or II piano key), the object's perspective distortion parameters are determined. For example, the perspective distortion parameters are determined based on the angle between the object's position in the image and the optical axis of the image acquisition module (i.e., the perspective angle). Positions closer to the optical axis have smaller perspective angles than those at the edges, resulting in lower distortion.

[0033] In this paper, morphological operations are a set of mathematical methods for processing digital images, primarily used to extract specific shape information from image components. Based on set theory principles, its main operations include dilation, erosion, opening, and closing operations, used to achieve functions such as noise removal, shape feature enhancement, or weakening. The structuring element, a core tool in mathematical morphology for image processing, is essentially a set of pixels with known shape, size, and orientation, defining the shape and scope of morphological operations.

[0034] The following detailed description, in conjunction with the accompanying drawings, outlines some embodiments of this application. Unless otherwise specified, the following embodiments and features described herein can be combined with each other. Please refer to... Figure 1 , Figure 1 This is a schematic flowchart illustrating a piano key recognition method provided in an embodiment of this application, such as... Figure 1 As shown in the figure, this application provides a piano key recognition method; the method includes S301 to S305.

[0035] S301, Obtain a first field-of-view image, which includes several images corresponding to a class of piano keys.

[0036] The first field-of-view image refers to the original image directly acquired by the image acquisition module. This image needs to contain a type of piano key that needs to be identified, serving as the input basis for subsequent piano key recognition.

[0037] Please see Figure 2 , Figure 2 This is a schematic diagram of a first field-of-view image provided in an embodiment of this application, such as... Figure 2 As shown, the musical instrument is a piano, and the image acquisition module is vertically positioned on the key plane and directly above the keyboard. The first field-of-view image includes several Class 1 keys 100 that need to be identified (the black keys in the image), and may also include other parts of the instrument, such as the casing, strings, hammers, etc., as well as the background of the venue and the performer's hand movements. It is evident that complex and interfering environmental factors exist in actual performance scenarios, such as... Figure 2 The piano keys reflect light, and the background is cluttered.

[0038] S302, perform a first filtering operation on the pixels in the first field of view image based on the pixel values ​​in the first field of view image to suppress the image noise of the first morphology and obtain a second field of view image.

[0039] Specifically, the first type of image noise refers to discrete, isolated interference in an image, such as random discrete noise generated by the sensor, dust reflections, or tiny light spots in the environment. This type of noise manifests in the image as interference points with significantly different pixel values ​​from the surrounding pixels but scattered spatial distribution, which may interfere with the integrity and continuity of critical structures such as piano key outlines.

[0040] Correspondingly, the first filtering operation is an image processing method that adaptively smooths local areas based on the grayscale or color distribution characteristics of each pixel in the first field of view image. For example, by analyzing the pixel value distribution in the pixel neighborhood, a multi-directional smoothing strategy is used to process the first field of view image, thereby suppressing image noise of the first morphology.

[0041] In some embodiments, selective smoothing is performed only on the first field-of-view image. For example, pixels suspected of being noise (such as those whose difference from the neighborhood median exceeds a preset median threshold) are replaced with neighborhood statistics (such as the median or weighted mean), while the original values ​​are retained for the edges of the piano keys.

[0042] In some embodiments, the multi-directional smoothing strategy can be median filtering, Gaussian filtering, or adaptive weighted filtering. Preferably, the multi-directional smoothing strategy is Gaussian filtering. It should be understood that Gaussian filtering uses a Gaussian function to allocate weights, with pixels closer to the center receiving higher weights and pixels farther from the center receiving lower weights. This weight allocation mechanism makes Gaussian filtering more natural and smooth in removing noise, but it cannot completely eliminate noise. Thus, some noise reduction is sacrificed to ensure the preservation of image details and piano key edges.

[0043] It should be understood that the first stage only performs local smoothing on discrete, isolated noise points (such as sensor noise or dust reflections). The multi-directional smoothing strategy based on pixel distribution characteristics maintains the overall image structure, avoids excessive denoising leading to detail loss, and ensures the geometric integrity of the piano keys, thus providing a reliable basis for parameter calibration in the second stage. Because the interference filtering intensity in the first stage is strictly limited, the resulting second-view image only initially removes discrete noise. While preserving the clarity of the main piano key structure, it also retains complex environmental interferences such as local breaks caused by shooting angle, uneven lighting, or black key textures. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of a second field-of-view image provided in an embodiment of this application, such as... Figure 3 As shown, from a macroscopic perspective, environmental interference such as piano key reflections and cluttered backgrounds still exist in the second field of view image.

[0044] In some embodiments, S302 includes: identifying at least one pixel curve on the first field-of-view image, the pixel curve being composed of a plurality of pixels and used to reflect the degree of change in pixel values ​​of the pixels; calculating the degree of pixel difference between at least one first target segment and an adjacent second target segment in the pixel curve, wherein the first target segment and the second target segment are composed of at least one pixel; determining whether the degree of pixel difference exceeds a pixel difference threshold; if so, correcting the first target segment to reduce the degree of pixel difference.

[0045] Specifically, at least one pixel curve in the first field-of-view image is identified. This pixel curve consists of continuously arranged pixels, and its pixel values ​​dynamically change along the curve path, such as through light and dark transitions or texture variations. Further, the pixel curve can be segmented with a fixed length or adaptively to obtain target segments of several consecutive pixels. For any first target segment, at least one adjacent second target segment is obtained, and the pixel difference degree of the target segments is calculated, such as mean difference, variance ratio, or maximum gradient difference. If the pixel difference degree exceeds a pixel difference threshold, noise regions exhibiting abrupt peaks or valleys are identified. The first target segment is then corrected, for example, by replacing it with a smooth transition value of the second target segment or by interpolation reconstruction, to eliminate abrupt changes and reduce the degree of difference. The pixel difference threshold is a preset difference tolerance threshold used to trigger the correction operation; its specific value can be determined based on noise feature statistics or empirical values ​​and is not limited here.

[0046] In some embodiments, if not, the original pixel values ​​of the first target segment are preserved to avoid redundant corrections to normal pixels, thereby maintaining the integrity of the key edges and details.

[0047] In some embodiments, multiple pixel curves are generated in the first field-of-view image along a specific direction, which may be a horizontal direction, a vertical direction, or the tangent direction of the piano key edge, etc., and are not limited here.

[0048] Therefore, by analyzing the pixel value differences between adjacent regions segment by segment, the target segments that exhibit continuous or step changes can be distinguished, the first-morphological noise can be accurately located and suppressed, and the blurring of the piano key outline due to over-correction can be avoided, providing a clearer intermediate image for subsequent thresholding.

[0049] S303, the second field of view image is processed by a preset threshold to obtain a third field of view image. The third field of view image includes a number of target pixel sets, which are composed of a number of consecutive pixels with pixel values ​​lower than the preset threshold.

[0050] The preset threshold processing is a step that converts the grayscale or color information of the second field of view image into a binary image. This involves dividing image pixels into foreground and background by setting a fixed threshold or a dynamically calculated adaptive threshold. Preferably, the preset threshold processing is adaptive binarization corresponding to an adaptive threshold. This is based on dynamically calculating the threshold according to the regional illumination, adapting to uneven illumination or reflective interference. Furthermore, an illumination normalization layer is added before adaptive binarization, and Contrast Limited Adaptive Histogram Equalization (CLAHE) is used to improve local contrast without amplifying noise, thus achieving adaptive illumination compensation.

[0051] Specifically, a preset threshold is applied to the second field-of-view image to obtain a binary third field-of-view image. Pixels in the third field-of-view image that are below the preset threshold are marked as target pixels, and adjacent target pixels form a target pixel set. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of a third-view image provided in an embodiment of this application, such as... Figure 4 As shown, taking black and white piano keys as an example, all continuous and adjacent "black" pixels (i.e. areas below a preset threshold) in the third field of view image form a target pixel set. Each black piano key will correspond to a target pixel set, enabling subsequent steps to more accurately identify the target pixel set as an independent target set, while eliminating minor interference.

[0052] It should be understood that threshold segmentation further clarifies the previously preserved key outlines. For example, it can sharpen slight edge blurring caused by smoothing in step S302 and suppress local grayscale interference caused by uneven lighting, initially closing microscopic fracture areas inside the black keys caused by texture or uneven lighting. At the same time, it enhances the contrast between the keys and the surrounding environment, such as further differentiating between black and white keys in a piano, providing high-confidence binarized input for subsequent morphological operations.

[0053] In some embodiments, before step S303, the method further includes: increasing the chromaticity-weighted grayscale of the second field-view image, i.e., enhancing the difference weights of the green and blue channels. It should be understood that, given the tendency towards black and white colors, at the boundary between black and white bonds, the average color difference between green and blue is significantly higher than that between red and green, or red and blue. Furthermore, under strong light, the chromaticity difference between black and white bonds may be more significant than the brightness difference. Therefore, enhancing the green and blue channels can maximize the color difference, thereby preserving more effective features in subsequent processing.

[0054] S304, perform a second filtering operation on the target pixel set in the third field-of-view image based on a preset structuring element to suppress image noise of the second morphology and obtain a fifth field-of-view image.

[0055] The preset structuring element is a pre-defined set of pixels used to detect and correct shape defects in the target pixel set. Its size and shape must be adapted to the physical characteristics of a type of piano key, such as the width of a type of piano key. Correspondingly, the second filtering operation is a morphological operation based on the preset structuring element. The main operations include dilation, erosion, opening, and closing operations, which are used to achieve functions such as noise removal, shape feature enhancement, or weakening.

[0056] Specifically, based on morphological operations, structured processing repairs second-morphological image noise in the third-view image by using preset structural elements. Second-morphological image noise refers to non-discrete, continuous interference in the image, which is structural noise caused by shooting conditions or physical structures. Examples include tiny gaps inside piano keys due to wood grain or uneven lighting, as well as linear interference caused by key reflections, key seams of non-standard keys, or dust accumulation.

[0057] It should be understood that the second stage fills in the microscopic fractures inside the black keys caused by the shooting angle through pixel aggregation, reconstructs the complete closed area, and then performs fine boundary trimming to simultaneously eliminate structural noise and background interference. This achieves simultaneous optimization of repairing effective features and stripping ineffective features, avoiding multiple broken edges generated by unclosed contours during subsequent extraction, which would lead to chaotic contour hierarchy relationships.

[0058] Please see Figure 5 , Figure 5 This is a schematic diagram of a fifth field-of-view image provided in an embodiment of this application, such as... Figure 5 As shown, the fifth field-of-view image after the second filtering operation eliminates burrs on the edges of piano keys (such as the bottom edge of black keys), key seams on white keys, and multiple linear interferences in the background, thus achieving refined boundary trimming. Simultaneously, the interior of the black keys appears as a completely closed, continuous state, as shown... Figure 4 As shown, the rightmost black piano key, due to interference from the reflective area, exhibits multiple breaks after the preset threshold processing. Figure 5 As shown, the gaps in the rightmost black piano key in the fifth field of view image caused by reflection have been corrected to a complete and continuous state. This avoids the formation of multiple broken edges in the unclosed contours during subsequent extraction, which would lead to confusion in the contour hierarchy. At the same time, the original geometric features of the piano key are preserved, meaning that the rightmost black piano key is still in a fragmented form.

[0059] It should be understood that, in order to ensure that the details of the piano keys are fully preserved, the intensity of interference filtering in the second stage is still strictly limited, for example... Figure 5 The presence of missing black keys and background interference in the image is due to the conservative parameter settings used in the second-stage interference filtering. This avoids excessive interference that could damage the original geometric features of the keys, providing highly complete and realistic contour data for subsequent perspective correction and physical verification. Furthermore, both the first and second stages are pre-processing steps, involving fine-tuning based on a high degree of respect for the original image data to eliminate specific image noise.

[0060] In some embodiments, the image noise of the second form includes burr noise and linear noise; S304 includes: expanding the boundaries of each target pixel set in the third field of view image based on a first structuring element to fill the internal holes or breaks of the target pixel set, thereby obtaining a fourth field of view image; and shrinking the boundaries of each target pixel set in the fourth field of view image based on a second structuring element to eliminate the burr noise of each target pixel set and the linear noise in the fourth field of view image, thereby obtaining the fifth field of view image.

[0061] Specifically, a boundary expansion operation is performed on the target pixel set in the third field-of-view image using the first structuring element. Morphological dilation fills the gaps between adjacent pixels to repair holes or breaks within the target pixel set caused by thresholding or uneven illumination, such as local missing areas in the black key texture. This results in the fourth field-of-view image, where the continuity of the target pixel set is enhanced, but the edges may exhibit slight redundancy due to the expansion. Further, a boundary contraction operation is performed on the fourth field-of-view image using the second structuring element. Morphological erosion eliminates burr noise at the edges of the target pixel set and residual linear noise in the image. In the final fifth field-of-view image, the contours of the target pixel set are smoother and more compact, while preserving the overall structural features of the piano keys. Burr noise refers to small, irregular protrusions caused by preset thresholding or uneven illumination, such as jagged defects on the edges of piano keys. Linear noise refers to thin, elongated stripe-like noise in the image caused by light interference or sensor errors, as well as other line interference in the background.

[0062] It should be understood that the preset structural elements include a first structural element and a second structural element. To ensure that the details of the piano keys are fully preserved, a more conservative first and second structural element are selected. For example, the design is tailored to the physical characteristics and noise features of a certain type of piano key, and the shape and size of its pixel set are strictly limited. Furthermore, the first and second structural elements can be set to different structures according to their different functions. The specific values ​​can be flexibly set according to the actual situation and are not limited here.

[0063] S305, based on the second preset model, identify several piano keys of the first type from the fifth field of view image.

[0064] Specifically, by inputting a purified set of target pixels (such as continuous, unbroken piano key regions) from the fifth field-of-view image, the second preset model, combined with a pre-trained feature library (such as piano key shape, size, and arrangement patterns) or rule-based algorithms (such as contour matching and proportion analysis), accurately locates and classifies a class of piano keys that conform to preset features. For example, if the target is a black key, the model will identify narrow, elongated, and regularly arranged dark regions in the image. It should be understood that since the preceding processing has significantly purified the image, the second preset model only needs to use simple rules or pre-trained feature matching to identify a class of piano keys from a continuous set of target pixels. Therefore, the second preset model is a pre-configured recognition algorithm or a trained classification model. Its feature library or rules are constructed based on prior information such as the geometry and arrangement of a class of piano keys, and can also use existing algorithms or contour matching models, etc., without limitation here. The identification result of a class of piano keys can be the piano key contour and / or the position coordinates of the piano keys in the image.

[0065] In some embodiments, because some noise or interference information is allowed to be transmitted to subsequent stages in the aforementioned processing steps, the extracted contours of a type of piano key may have problems such as local defects or uneven contours. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of a type of piano key contour extraction provided in an embodiment of this application, such as... Figure 6 As shown, the red lines represent the extracted key outlines of one type. Due to the significant effect of the preceding denoising stage, the second preset model accurately and completely extracted each key of this type (the black keys in the figure). However, the outlines of many black keys are not smooth; some keys (such as the rightmost black key) are not correctly identified in certain areas. Based on this, this application proposes a contour correction mechanism based on perspective deformation.

[0066] In some embodiments, the method further includes: determining the perspective deformation parameter of each type of piano key based on preset camera parameters and the position coordinates of the type of piano key in the fifth field of view image; determining the target shape parameter of each type of piano key based on the reference shape parameter of the type of piano key and the perspective deformation parameter; and adjusting the contour of the corresponding type of piano key based on the target shape parameter of each type of piano key.

[0067] Specifically, based on preset camera parameters and the position coordinates of a type of piano key in the fifth field of view image, the perspective deformation parameters (such as shrinkage ratio) of each type of piano key are determined. The baseline shape parameters of the piano keys are then adjusted according to these parameters to simulate a perspective effect, causing the baseline shape parameters to deform due to perspective, such as edge stretching or shortening. Finally, the piano key contours are corrected based on the target shape parameters of each key to ensure that the calibrated key contours reproduce the deformation errors caused by the shooting angle, thereby improving the accuracy of subsequent positioning.

[0068] It should be noted that for the same preset camera parameters, the perspective distortion parameters at each position coordinate are determined. Based on the preset camera parameters and the position coordinates of a class of piano keys, the corresponding perspective distortion parameters can be quickly obtained. The perspective distortion parameters at each position coordinate in the image are determined based on prior data and are not limited here. Furthermore, the target shape parameters can be derived through affine transformation or projection transformation to obtain the target shape parameters of each piano key.

[0069] In some embodiments, the reference morphological parameters can be the geometric feature parameters of the piano keys in the image without perspective distortion, such as width, length, aspect ratio, etc. The reference morphological parameters can be obtained in advance, or the morphological parameters of the piano keys with the lowest degree of perspective distortion can be determined from the image as the reference morphological parameters.

[0070] In some embodiments, the method further includes: acquiring at least one proximal type of piano key with a perspective deformation parameter lower than a first deformation threshold, and determining a reference morphological parameter of the type of piano key based on the morphological parameters of the at least one proximal type of piano key; acquiring at least one distal type of piano key with a perspective deformation parameter higher than a second deformation threshold, wherein the first deformation threshold is less than or equal to the second deformation threshold; adjusting the reference standard morphological parameter of the type of piano key based on the perspective deformation parameter of each distal type of piano key, and determining a target morphological parameter for each distal type of piano key; and adjusting the contour of the corresponding distal type of piano key according to the target morphological parameter of each distal type of piano key.

[0071] Specifically, a category of near-field keys (such as those directly in front of the camera with almost no perspective distortion) with perspective distortion parameters below the first distortion threshold are selected, and their shapes closely resemble the baseline shape under a standard viewpoint. By statistically analyzing the shape parameters of at least one near-field key, the baseline shape parameters of this category of keys are calculated and updated as a reference for subsequent adjustments. For a category of far-field keys with perspective distortion parameters above the second distortion threshold (such as keys at the edge of the image that are significantly distorted due to tilt), the baseline shape parameters are dynamically adjusted based on their specific perspective distortion parameters through perspective transformation. For example, if a key's width is shortened due to tilting at a distance, the baseline shape parameters are adjusted to simultaneously shorten the width, generating the target shape parameters. The first distortion threshold is used to identify near-field keys less affected by perspective distortion. Correspondingly, the second distortion threshold is used to identify far-field keys affected by perspective distortion; the specific values ​​can be flexibly set and are not limited here.

[0072] Based on the target shape parameters of each distal key, its contour shape in the image is adjusted (such as stretching or scaling pixel areas in a specific direction) to make it closer to the shape after perspective. Thus, through regional processing of near and far ends and dynamic parameter correction, the key deformation caused by the difference in camera viewpoint is adapted in the contour calibration to accurately locate the keys at various positions in the image.

[0073] In some embodiments, when no perspective occurs, the scaling ratio in the perspective deformation parameter is 1.0 times that of the target shape parameter and the reference shape parameter. As a type of piano key gradually moves away from the optical axis in the image, the scaling ratio changes linearly and gradually reaches 0.5 times that of the target shape parameter and the reference shape parameter.

[0074] In some embodiments, the contours of several types of piano keys are obtained, and the keys are filtered based on the width of the contours of the types of keys to obtain keys with a width greater than a width threshold. The narrowest type of keys is then selected, and its shape parameters are used as the baseline shape parameters. The width threshold can be flexibly set and is not limited here.

[0075] In some embodiments, the target morphological parameter is set with a limit value. When the target morphological parameter obtained by adjusting the reference morphological parameter is less than the limit value, the limit value is taken as the target morphological parameter. And / or, the number of abnormal occurrences where the target morphological parameter obtained by adjusting the reference morphological parameter is less than the limit value is counted. If the number of abnormal occurrences is greater than a preset number of abnormal occurrences, a new type of proximal key is selected to update the reference morphological parameter. For example, if key A is selected as a type of proximal key, and the target morphological parameter obtained by adjusting the reference morphological parameter of key A is less than the limit value, the limit value is taken as the target morphological parameter. If the target morphological parameter is less than the limit value multiple times consecutively, key B is selected as a type of proximal key.

[0076] It should be understood that in the third stage, by combining the morphological data of the proximal keys as the reference parameter and processing the distal keys through perspective deformation parameters, the reliability of parameter calibration is maintained, while avoiding parameter distortion caused by perspective projection. Even when there are differences in instrument manufacturing processes, changes in instrument models, or changes in shooting angles, a high recognition rate can still be maintained, and a stronger generalization ability can be achieved.

[0077] In some embodiments, please refer to Figure 7 , Figure 7 This is a schematic diagram of another type of piano key wheel extraction provided in the embodiments of this application, such as... Figure 7As shown, the red lines represent the extracted outlines of one type of piano keys. The second preset model accurately extracts each key of this type without omission. However, due to strong background interference, the second preset model mistakenly identifies non-key areas as keys, such as the striped carpet and long, narrow frames in the image. At this point, position verification or arrangement pattern analysis in the fourth stage can further filter out misjudgments, ensuring the accuracy of the final result. Therefore, the extraction of one type of piano keys in step S305 is also a form of filtering for image interference. The filtering strength is still strictly limited at this stage. High-quality fifth-view images are used for efficient initial screening, rather than pursuing absolute accuracy, tolerating a certain degree of recognition error.

[0078] In some embodiments, the method further includes: obtaining the actual shape parameters and position coordinate parameters of each type of piano key; based on preset piano key layout parameters, filtering a number of the type of piano keys according to the actual shape parameters and position coordinate parameters of the type of piano keys to determine a number of target type of piano keys; and outputting the position coordinates of the number of target type of piano keys in the first field of view image.

[0079] The preset key layout parameters are a standardized set of physical parameters for a type of piano key in the image, used to verify the geometric and arrangement rationality of the recognition results. For example, the preset key layout parameters include at least one of the following: target length, target width, target aspect ratio, target area, target key spacing, and target number of keys. For instance, a standard piano has 36 black keys.

[0080] Specifically, for each identified type of piano key, or each type of piano key whose outline has been adaptively corrected based on perspective deformation, its actual shape parameters and position coordinate parameters are obtained. Multi-level verification is then performed based on preset piano key layout parameters. First, combining preset piano key layout parameters (target length, target width, target aspect ratio, target area, etc.), the shape of individual keys is verified, strictly selecting areas that conform to the physical characteristics of the keys, eliminating types of keys whose size, aspect ratio, or area deviates significantly, and removing interference objects with abnormal sizes. Second, the overall layout of the keyboard is verified through key spacing, number, and arrangement pattern, retaining types of keys that conform to preset arrangement rules and key quantities, ensuring that the recognition results conform to the standard structure of musical instruments. Thus, the target type of piano keys that conform to the standards are accurately retained, and misidentified non-key areas in the background are eliminated. The precise position coordinates of the target type of piano keys in the original first-view image are output, providing reliable data for subsequent positioning or interactive operations.

[0081] In this embodiment, the interference filtering intensity in step S302 is strictly limited, only initially purifying discrete noise. While preserving the clarity of the main key structure, it also retains complex environmental interferences such as local breaks caused by shooting angle, uneven lighting, or black key texture. The removal of discrete isolated noise points is strictly limited to be completed before thresholding (step S303) to ensure that edge blurring caused by the smoothing strategy is clearly restored during thresholding, ensuring the geometric integrity of the keys and providing a high-confidence binary input for subsequent morphological operations. Furthermore, step S304, while removing continuous noise, repairs key or edge breaks that occur after thresholding, ensuring closed key contours and preventing multiple broken edges from unclosed contours during subsequent extraction, which could lead to chaotic contour hierarchy. Because the interference filtering in the second stage uses conservative parameter settings, the processed keys may still have defects, and the background may still contain interference.

[0082] In step S305, the extraction of a type of piano key utilizes a high-quality fifth-view image for efficient initial screening, filtering out most background interference. However, the filtering intensity is still strictly limited at this stage; absolute accuracy is not pursued, and a certain degree of recognition error is tolerated, such as contour defects of the piano key type or misjudgment of non-key areas. Please refer to [link / reference]. Figure 8 , Figure 8 This is a schematic diagram of a type of piano key with adjusted outline provided in an embodiment of this application, such as... Figure 8 As shown, these recognition errors are eliminated by adaptively correcting the piano key contours based on perspective deformation and by performing hard physical verification based on the piano key layout parameters. Figure 8 The green lines represent the adjusted outline of one type of piano key. While accurately conforming to this type of key, the outline presents a smooth and standardized shape. Furthermore, to avoid premature pruning that could lead to the accidental deletion of effective outlines due to localized deformation, physical verification is performed at the end of the process.

[0083] Therefore, the various steps in this application embodiment have a high degree of temporality and coupling. Through a phased and progressive interference filtering mechanism, even if some noise or ambiguous information is allowed to be transmitted to subsequent stages, it will be gradually identified and corrected in subsequent steps. In the end, high-precision interference removal is achieved while preserving the details of the piano keys, which effectively improves the recognition accuracy and robustness of piano keys in complex scenarios.

[0084] Please see Figure 9 , Figure 9 This is a schematic flowchart illustrating a piano key recognition method based on key seams, as provided in an embodiment of this application. Figure 9 As shown, this application provides a piano key recognition method based on key seams; the method includes S401 to S404.

[0085] S401, acquire a first field-of-view image, the first field-of-view image including several images corresponding to a type I piano key and a type II piano key.

[0086] The first field-of-view image refers to the original image directly acquired by the image acquisition module. This image simultaneously presents one type of piano key and two types of piano keys. It needs to contain sufficient details to support subsequent localization and segmentation operations, such as clear key seam boundaries or piano key outlines, as the input basis for subsequent piano key recognition.

[0087] S402, based on the preset positional relationship between the first type of piano keys and the second type of piano keys, according to the piano key image area where the first type of piano keys are located, a number of first partial images of the areas where the second type of piano keys are located are extracted from the first field of view image.

[0088] Among them, the preset positional relationship refers to the fixed spatial arrangement rules between the first type of piano keys and the second type of piano keys. Taking the piano as an example, the black keys are usually located above the gap of the adjacent white keys. There is a fixed size ratio and a specific interval arrangement between them, such as two black keys sandwiched between three white keys.

[0089] Specifically, based on the known image regions of one type of piano key, and according to preset positional relationships, a first local image containing only two types of piano keys is extracted from the first field-of-view image. This rapidly narrows the processing scope through spatial correlation, converging global processing to a local level, reducing computational load and eliminating irrelevant background interference, while preserving the complete details of the target region, providing high-precision input for subsequent key seam recognition.

[0090] Please see Figure 10 , Figure 10 This is a schematic diagram of a first partial image cropping provided in an embodiment of this application, such as... Figure 10 As shown, the musical instrument is a piano, and the image acquisition module is vertically positioned on the key plane and directly above the keyboard. The first field-of-view image includes several Class I keys 100 (black keys in the figure) and several Class II keys 200 (white keys in the figure). According to the preset positional relationship, a rectangular area is cropped below and to both sides of the Class I keys 100, as shown by the red dotted line area, to cover the area of ​​any possible Class II keys 200.

[0091] Please see Figure 11 , Figure 11 This is a schematic diagram of a key recognition method based on key seams provided in an embodiment of this application, as shown below. Figure 11As shown in Figure 40a, the Region of Interest (ROI) of the white keys is extracted, which is the first local image. This includes local images of several second-class keys, quickly eliminating irrelevant background interference. It should be understood that there are significant differences in color, shape, or texture between first-class and second-class keys, such as the contrast between dark and light colors. Therefore, the first local image can be quickly extracted from the first-class keys.

[0092] In some embodiments, regions matching the size and shape of a certain type of piano key can be selected by combining shape features (such as rectangles and aspect ratios); alternatively, template images (such as local textures or contours) of a certain type of piano key can be pre-stored, and sliding matching can be performed in the target image to find the region with the highest similarity; alternatively, a detector for a certain type of piano key can be trained using a large amount of labeled data to output the position and category. The features of a certain type of piano key are more obvious than those of a second type of piano key, and its recognition accuracy is higher. The recognition method for a certain type of piano key can be found in related technologies and is not limited here.

[0093] In some embodiments, a piano key recognition method provided in any embodiment of this application can be used to identify a type of piano key and its position coordinates in a first field-of-view image, thereby determining the piano key image region where the type of piano key is located.

[0094] In some embodiments, piano keys in common musical instrument keyboards are divided into two categories: black keys and white keys. Black keys are considered as one type of key, and white keys as another. Preferably, a first local image is extracted based on the coordinates of the black keys, and then white key recognition is performed based on this first local image. Compared to white keys, the features of black keys are more obvious. Global processing can avoid introducing a lot of noise and improve the efficiency of calculation. At the same time, the projection of black keys and the edges of white keys are difficult to distinguish in the global image, and local processing of white keys can reduce the false detection rate.

[0095] In some embodiments, a preset positional relationship is determined based on prior knowledge of standard musical instruments. For example, the preset positional relationship between the first type of piano keys and the second type of piano keys is the shape ratio or area ratio of the first type of piano keys and the second type of piano keys. The shape ratio refers to the relative geometric shape of the two types of keys, such as the ratio of length, width, or height; for example, the length of a white key in a piano is generally three-half the length of a black key. The area ratio refers to the ratio of the areas of the two types of keys in an image or actual physical structure.

[0096] In some embodiments, after S402, the method further includes: performing a preset threshold processing on the first local image and executing a first filtering operation to suppress image noise of the first form; or, performing a first filtering operation on the first local image to suppress image noise of the first form and performing preset threshold processing.

[0097] Specifically, the preset threshold processing is a binarization process corresponding to a global or adaptive threshold, separating the piano keys in the first local image from other background elements (such as key seams); the first filtering operation is to suppress image noise of the first morphology through a multi-directional smoothing strategy, preferably a Gaussian filtering strategy. Specific implementation methods can be found in the foregoing embodiments and will not be repeated here. The specific parameter settings for the preset threshold processing and the first filtering operation can be the same as or different from those in the foregoing embodiments, and are not limited here. Furthermore, since the background interference in the first local image is extremely low, and the key seam features are clear and not easily blurred, the two processing methods can be flexibly combined in sequence, and are not limited here. The processed first local image is as follows: Figure 11 As shown in Figure 40b.

[0098] S403, identify several initial key gaps between adjacent two types of piano keys in the first local image; and perform morphological operations on the several initial key gaps in the first local image to obtain the target local image.

[0099] Specifically, the initial key gaps between adjacent types of piano keys, such as low-brightness areas or boundary lines, are identified from the first local image. Figure 11 In Figure 40c, the initial inter-bond gap identification result is shown, where the lines corresponding to the white pixels represent the initial inter-bond gaps. Further, morphological operations are performed on these initial inter-bond gaps to eliminate structural noise and normalize the gap morphology. For example... Figure 11 The image in 40d is the target local image output after morphological operation processing. The gaps between the keys have been corrected to continuous strips that conform to physical laws (such as uniform width and perpendicular direction). At the same time, the arrangement relationship between the key gaps and the piano keys is preserved, providing a high-confidence input for the piano key region extraction of S404.

[0100] It should be understood that the key gap (also known as the key gap) is the natural boundary region between two adjacent types of piano keys in the first local image, appearing as a narrow strip with low brightness or high contrast. Its identification methods include local feature analysis based on color contrast (such as the brightness difference between the key gap and the piano keys) or edge detection. Furthermore, as... Figure 11 As shown in Figure 40c, the image region corresponding to the initial inter-bond gap is very narrow. Morphological processing of this region requires strict limitation of the corresponding structural elements to avoid overfilling of the gap or destruction of the original gap information.

[0101] In some embodiments, S403 includes: identifying the outlines of several types of piano keys in the first partial image based on the color information of several pixels in the first partial image; identifying several candidate key gaps between the two types of piano keys in the first partial image; obtaining at least one neighboring type of piano key adjacent to the candidate key gap; and determining the gap tilt threshold of the neighboring type of piano keys based on the longitudinal outline of the neighboring type of piano keys according to a preset angle constraint; when the tilt angle of the candidate key gap meets the gap tilt threshold, taking the corresponding candidate key gap as the initial key gap.

[0102] Specifically, based on the color features of pixels in the first local image, such as grayscale values ​​or color saturation, narrow lines with abrupt brightness changes are detected as candidate key gaps. These candidate key gaps include not only real key gaps but also pseudo gaps caused by reflections, stains, or perspective distortion, such as oblique bright lines formed by reflections. Neighboring keys of two types are acquired near the candidate key gaps, such as keys to the left and / or right of the candidate key gap, or keys within a preset adjacent range of the candidate key gap. The perspective distortion of the candidate key gaps and neighboring keys of two types is related and similar. The actual arrangement direction of the longitudinal contours of the neighboring keys of two types is analyzed, and a tolerable gap tilt threshold is calculated based on preset angle constraints. Several candidate key gaps are then filtered, retaining the initial key gaps whose tilt meets the requirements, while those exceeding the threshold are judged as noise or pseudo gaps and excluded.

[0103] The preset angle constraint is a pre-set tolerable angle deviation. The specific value can be flexibly set according to actual needs, such as 15°, and is not limited here.

[0104] The longitudinal profile refers to the edge profile of the two types of piano keys extending along their long axis, such as the straight edges on both sides of the white keys. It is used to determine the relative angle between the main extension direction of the keys and the image coordinate system. It should be understood that the longitudinal profile of the two types of piano keys may be tilted due to the camera's pitch angle or perspective. Therefore, dynamically calculating the gap tilt threshold based on the actual longitudinal profile direction can accurately match the key gap direction of the actual two types of piano keys in the image. For example, if the actual longitudinal profile direction is tilted 5° to the right, combined with a preset angle constraint of 15°, the gap tilt threshold is tilted from 10° to 20° to the left and candidate key gaps within this tilt angle range are used as the initial key gaps.

[0105] Therefore, by combining the key outline direction and dynamic angle constraints, it is ensured that the identified key gaps conform to the actual tilt angle of the keys in the image, and only the gaps between keys that conform to the perspective rules of specific image positions are retained, effectively eliminating interference information and providing a precise data foundation for subsequent key gap repair and key positioning.

[0106] In some embodiments, a vertical edge enhancement strategy is employed based on the geometric characteristics of the parallel arrangement of two types of piano keys. A customized horizontal gradient filter prioritizes capturing the vertical gap features between the two types of keys while suppressing oblique interference edges. The image gradient distribution is dynamically analyzed, and high and low thresholds are automatically set to ensure a balance between weak edge preservation and noise suppression. Simultaneously, the detected edges (i.e., candidate gaps between keys) undergo directional filtering, retaining only valid edges that deviate from the arrangement direction of the two types of keys by less than 15°, and selecting approximately vertical lines. This angular constraint eliminates oblique interference such as key projection and reflection, ensuring the physical consistency of the edges.

[0107] In some embodiments, the morphological operation includes the step of: S4031, obtaining at least one target type II piano key adjacent to the initial key gap to be processed, and updating the setting parameters of the structural element based on the perspective deformation parameters of the target type II piano key.

[0108] Specifically, for each initial key gap to be processed, or for multiple initial key gaps to be processed within a preset shared range, specific structural element setting parameters are used. By analyzing at least one target type II piano key adjacent to the initial key gap, the perspective deformation parameters of the target type II piano key are obtained. The setting parameters of the structural elements used in the morphological operations are dynamically calculated and updated to match the actual deformation characteristics of the key gap. It should be understood that piano keys exhibit a tilted or longitudinally contracted state due to perspective, causing the direction of their adjacent key gaps to deviate from the standard horizontal direction. In this case, the structural elements are adaptively adjusted according to the specific perspective deformation of each key to differentiate the processing of different key gaps.

[0109] In some embodiments, the difference in perspective deformation parameters of multiple initial key gaps within a preset shared range is less than the preset deformation difference, and they can share the setting parameters of a single structural element. The specific range division can be predetermined and is not limited here.

[0110] In some embodiments, the target type II key can be the type II key on the left and / or right side of the initial key gap. When multiple initial key gaps share the setting parameters of a single structural element, the type II key on the left and / or right side of any one of the initial key gaps can be selected as the target type II key.

[0111] For example, the long axis of the structural element needs to be aligned with the direction of the adjacent target type II piano keys so that the processed key seam matches the shape of the adjacent target type II piano keys. The size parameters of the structural element can be adjusted according to the degree of piano key deformation, such as the length of the structural element shortening with perspective shrinkage.

[0112] It should be understood that the dynamic coupling mechanism based on perspective deformation parameters and structural element parameters enables morphological operations to accurately repair key seam breaks or eliminate noise, avoiding processing deviations caused by fixed structural elements, thereby generating target local images that better conform to the deformation rules of piano keys in the image.

[0113] In some embodiments, the position coordinates of the target type II piano key in the first field-of-view image are obtained, and the perspective deformation parameters of the target type II piano key are determined based on preset camera parameters and the position coordinates of the target type II piano key. For example, the position coordinates of the target type II piano key in the first field-of-view image are obtained, and its perspective deformation parameters are calculated in combination with preset camera parameters. Alternatively, for the same preset camera parameters, the perspective deformation parameters at each position coordinate are determined, and the corresponding perspective deformation parameters can be quickly obtained based on the preset camera parameters and the position coordinates of the target type II piano key. The perspective deformation parameters at each position coordinate in the image are determined based on prior data and are not limited here.

[0114] In some embodiments, the perspective distortion parameters include a shrinkage ratio and / or a shrinkage direction. The shrinkage ratio refers to the specific numerical proportion of the deformation of an object (such as a type I piano key or a target type II piano key) in the image due to perspective distortion; for example, the shrinkage ratio in the length direction or the shrinkage ratio in the width direction compared to the unperspective image. The shrinkage direction refers to the direction in which the object (such as a type I piano key or a target type II piano key) shortens in the image due to perspective distortion. For example, because the farthest piano key is farther from the camera, its longitudinal direction appears shorter in the image; that is, the shrinkage direction is the depth direction perpendicular to the line of sight.

[0115] In some embodiments, the setting parameters of the structural element include the major axis direction, the major axis length, and the minor axis length. The method further includes: obtaining the perspective deformation parameters of the target type II piano key, the perspective deformation parameters including the shrinkage ratio and / or the shrinkage direction; determining the major axis direction of the structural element based on the shrinkage direction; and / or, determining the target shape parameters of the target type II piano key according to the standard shape parameters of the type II piano key and the shrinkage ratio; and determining the major axis length and the minor axis length of the structural element according to the target shape parameters of the target type II piano key.

[0116] Specifically, the long axis direction of the structural element is dynamically adjusted by the contraction direction of the target type II piano keys. If the keys contract longitudinally due to perspective, the contraction direction is the tilt direction of the key's long axis, which is also the tilt direction of the key seam. The long axis direction of the corresponding structural element must be strictly aligned with this contraction direction to ensure that the morphological operation is consistent with the actual direction of the key seam in the image. For example, if the distal keys show a shortening trend in the image and the keys tilt to the upper right, then the long axis of the structural element is rotated in the same direction.

[0117] Specifically, standard morphological parameters of the two types of piano keys are obtained, such as their original length and width before perspective distortion. These are then combined with the shrinkage ratio to calculate their target morphological parameters, such as their actual length and width after perspective shortening. Furthermore, the major axis length of the structural element is dynamically scaled based on the actual length in the target morphological parameters. And / or, the ratio between the two types of piano keys and the key gap width is obtained and set. The expected width of the key gap is determined based on the actual width and the ratio set in the target morphological parameters. The minor axis length of the structural element is dynamically scaled based on the expected width of the key gap, ensuring a high degree of match between the shape of the structural element and the physical characteristics of the actual key gap. Thus, by combining perspective distortion parameters with standard morphological parameters, the dimensional parameters of the structural element are precisely adapted to the shape of the piano keys from the current perspective, thereby improving the accuracy of key gap repair and noise suppression.

[0118] In some embodiments, the morphological operation further includes the step: S4032, performing morphological operations on the initial inter-bond gaps to be processed based on the updated structuring element.

[0119] Specifically, using the updated structural elements, morphological operations are performed on multiple initial key gaps within one or a pre-defined shared range to optimize the gap morphology. Through dynamically adapted morphological processing, the structural elements can accurately conform to the actual key gap direction and deformation patterns, preserving effective features while suppressing noise. As shown in 11.40d, the morphological operation thickens, expands, and dilates the white key edges (i.e., the initial key gaps), ultimately generating a target local image containing only complete, continuous, and physically consistent key gaps, improving the robustness and accuracy of subsequent key positioning.

[0120] In some embodiments, the method further includes: expanding the boundary of the initial inter-bond gap based on the updated structural elements to fill the internal holes or fractures of the initial inter-bond gap, thereby obtaining a second local image; and shrinking the boundary of the initial inter-bond gap in the second local image based on the updated structural elements to eliminate burr noise and linear noise of the initial inter-bond gap, thereby obtaining the target local image.

[0121] Specifically, the keyway features are optimized through two-stage morphological operations. First, based on the updated structuring element, an expansion operation, such as morphological dilation, is performed on the boundary of the initial keyway. This fills the internal holes or fracture areas in the keyway by sliding the structuring element, generating a second local image that ensures the continuity of the keyway. Second, the structuring element is used again to perform a contraction operation, such as morphological erosion, on the keyway boundary of the second local image. This eliminates edge expansion or pseudo-keyways caused by excessive expansion, ultimately forming a target local image that retains only complete, smooth, and physically consistent keyways.

[0122] In some embodiments, the initial key gaps are traversed based on the updated structuring element to expand the pixel set that matches the structuring element and / or shrink the pixel set that does not match the structuring element. For pixel sets that match the shape of the structuring element, such as those whose orientation matches the key gap tilt angle and whose size matches the perspective scaling ratio, an expansion operation is performed to repair key gap breaks or key adhesion caused by image noise or viewing angle tilt. For pixel sets that do not match the structuring element, such as those whose orientation does not match the key gap tilt angle, a shrinkage operation is performed to eliminate redundant or deviated areas, such as false gaps formed by edge burrs or background interference.

[0123] In some embodiments, the structural element is elliptical in shape and, as the major axis changes, presents a tilted elliptical form, i.e., a tilted elliptical kernel. For example, the tilted elliptical kernel is an asymmetric, directionally adjustable morphological structural element. It is elliptical in shape, and the lengths of its major and minor axes are dynamically calculated based on the actual physical dimensions of the type-II piano keys and camera parameters. The direction of the major axis is consistent with the perspective contraction direction of the type-II piano keys, enhancing the continuity of the key seams. The kernel parameters change with the image position to match the degree of deformation in different regions.

[0124] In some embodiments, a mathematical model of the kernel size is constructed based on the actual physical size of the second type of piano keys (approximately 23mm wide) and preset camera parameters (such as focal length and tilt angle). In the image edge deformation area, a tilted elliptical kernel is used to connect the gaps, with its major axis direction consistent with the perspective contraction direction of the white keys, to achieve geometric compensation.

[0125] It should be understood that the closer the piano keys are to the sides of the image, the smaller the key gaps, and under high light intensity, two white keys may stick together. The essential process of morphological analysis is dilation and erosion. In the embodiments of this application, the kernel that determines the degree of dilation and erosion is dynamically updated to avoid using the same kernel for processing, thereby avoiding the kernel being too large and shrinking the white key judgment area, or the kernel being too small and unable to separate the sticking white keys.

[0126] It should be understood that the introduction of a coupling mechanism between dynamic structural elements and perspective deformation parameters allows for real-time adjustment of the structural element's morphological parameters (such as the major axis direction and major and minor axis lengths) based on perspective deformation parameters like the key's tilt angle and contraction direction. This ensures precise matching with the key seam orientation and morphological changes, effectively improving the accuracy and anti-interference capabilities of key recognition. For example, if the key seam orientation deviates from the vertical direction due to camera tilt, the major axis direction of the structural element is adjusted to match the key seam tilt angle, ensuring morphological operations are performed along the key seam deformation direction and accurately repairing breakage or peeling noise. If the key seam width is scaled due to perspective projection, the size of the structural element is reduced proportionally to avoid overfilling or erosion, enabling the structural element to adapt to key seam deformation under different shooting conditions and maintain geometric consistency between near and far key seams, thereby improving the robustness of morphological operations.

[0127] S404, based on the third preset model, identify several of the second type of piano keys from the target local image.

[0128] Specifically, by inputting complete, continuous, and noise-free key gaps in the target local image, clear key boundary features are provided for the third preset model. The third preset model, combined with a pre-trained feature library (such as key shape, size, and arrangement patterns) or rule-based algorithms (such as contour matching and proportion analysis), accurately locates and classifies two types of keys that conform to the preset features. For example, if the target is a white key, a wide, continuous bright-colored region is extracted. Figure 11 As shown in 40e, the image regions of several second-class piano keys extracted from the target local image are compared and displayed in the first local image. The red lines are the extracted outlines of the second-class piano keys. It can be seen that the third preset model accurately and without omissions extracts each second-class piano key.

[0129] It should be understood that because the key seams are repaired by dynamic structural elements to remove fractures, eliminate burrs, and strictly match the perspective deformation of the piano keys, the highly standardized boundary contours of the key seams mean that the boundary contours of the piano keys are also highly standardized. The third preset model can identify two types of piano keys from the target local image using only simple rules or pre-trained feature matching. Therefore, the third preset model is a pre-configured recognition algorithm or a trained classification model. Its feature library or rules are constructed based on prior information such as the geometry and arrangement of one type of piano keys. Existing algorithms or contour matching models can also be used, and no restrictions are placed here.

[0130] In some embodiments, the method further includes: acquiring at least one proximal inter-bond gap with a perspective deformation parameter lower than a first deformation threshold, and determining a reference morphological parameter of the inter-bond gap based on the morphological parameter of the at least one proximal inter-bond gap; acquiring at least one distal inter-bond gap with a perspective deformation parameter higher than a second deformation threshold, wherein the first angle threshold is less than or equal to the second angle threshold; adjusting the reference morphological parameter of the inter-bond gap based on the perspective deformation parameter of each distal inter-bond gap, and determining a target morphological parameter for each distal inter-bond gap; and adjusting the contour of the corresponding distal inter-bond gap according to the target morphological parameter of each distal inter-bond gap.

[0131] Specifically, near-end key gaps (such as gaps between two types of piano keys directly in front of the camera with almost no perspective distortion) whose perspective distortion parameters are below the first distortion threshold are selected, as their shapes are close to the baseline shape under a standard viewpoint. The baseline shape parameters of the key gaps are calculated and updated by statistically analyzing the shape parameters of at least one near-end key gap, serving as a reference for subsequent adjustments. For far-end key gaps (such as gaps between two types of piano keys where the edge of the image is significantly distorted due to tilt), the baseline shape parameters are dynamically adjusted through perspective transformation based on their specific perspective distortion parameters. For example, if the length of the far-end key gap is shortened due to perspective, the baseline length is scaled proportionally; if the direction is offset, the major axis direction is adjusted to match the tilt angle.

[0132] Based on the adjusted target morphology parameters, the contour of the distal keyway is adjusted, such as... Figure 11 As shown in Figure 40f, the green lines represent the adjusted outline of the second type of piano keys. While accurately conforming to the second type of piano keys, the outline presents a smooth and standardized shape. This ensures that its shape is consistent with the baseline features of the near-end key seams, improving the geometric standardization and recognition accuracy of the overall key outline, while also adapting to the deformation patterns caused by its own viewing angle. Consequently, the second type of piano keys extracted subsequently can also adapt to the deformation patterns caused by their own viewing angle.

[0133] In this embodiment, a first type of piano key with significant features is identified first. Based on the spatial correlation between the keys (such as shape or area ratio), the local image of the second type of keys to be identified is quickly locked, converging the global view to the local image. This significantly reduces the image processing range, lowers computational complexity, and eliminates background interference while ensuring positioning accuracy. Furthermore, the natural boundaries between the keys—key seams—are selected as the core processing unit. A reverse normalization strategy avoids directly processing large homogeneous areas, further narrowing the image processing range to the microstructure of the key seams and reducing computational complexity.

[0134] In some embodiments, the cropping of the first local image is preferably performed before the first filtering operation and edge detection (i.e., initial key gap detection) to avoid unnecessary signal-to-noise ratio degradation. Secondly, morphological operations are preferably performed after edge detection, utilizing edge intensity maps to enhance feature connectivity and reduce the impact of isolated noise, thereby avoiding an increase in key gap breakage rate. Therefore, a progressive path of edge detection, morphological operations, and contour extraction (i.e., recognition of two types of piano keys) is adopted to progressively strengthen features. Finally, physical verification (such as spacing checks and aspect ratio filtering) is performed at the end of the process. At this point, judgment is based on complete contour information, preventing premature pruning and the accidental deletion of valid contours due to local deformation or information fragmentation.

[0135] In some embodiments, please refer to Figure 12 , Figure 12 This is a schematic diagram of a piano key recognition result provided in an embodiment of this application, such as... Figure 12 As shown in the embodiments of this application, a piano key recognition method can be used to recognize one type of piano key (the black keys in the figure), and the recognition result is shown in the red lines. A piano key recognition method based on key seams, also provided in the embodiments of this application, can be used to recognize two types of piano keys (the white keys in the figure), and the recognition result is shown in the green lines. Furthermore, the recognition result can be the position coordinates of several one-type piano keys and several two-type piano keys, the key outline, and the piano key image region in which they are located.

[0136] In some embodiments, in the step of determining the target shape parameters of each type of piano key based on the reference shape parameters and the perspective distortion parameters, and adjusting the contour of the corresponding type of piano key according to the target shape parameters of each type of piano key, the top surface position of the type of piano key is determined based on the position coordinates of the corresponding type of piano key in the image. Based on the top surface position, the contour of the corresponding type of piano key is adjusted according to the target shape parameters of each type of piano key, so that the contour of the corresponding type of piano key includes the top surface position. It should be understood that the difference in the three-dimensional height of the black keys causes perspective distortion and edge occlusion effects during imaging, which in turn causes their top surface to shift, such as... Figure 6 As shown, the rightmost black piano key exhibits a top-surface offset issue. Based on its position coordinates, its top surface position is predicted to be on the right side of the outline. The outline is then adjusted using the right side as the boundary. That is, if the outline needs to be reduced (e.g., adjusted to the expected width), the left side of the outline will be reduced first, while the right side of the image will be preserved. Figure 12 As shown, the adjusted outline prioritizes the right-hand image, ensuring accurate selection of the top surface of the piano keys. This provides reliable data for subsequent positioning or interactive operations, improving adaptability in complex technique analysis, teaching feedback, and intelligent musical instrument interaction.

[0137] For example, the above method can be implemented as a computer program that can run on a smart musical instrument. The smart musical instrument includes piano keys for direct interaction with the performer and playing music; an image acquisition module for acquiring a first field-of-view image, which includes images corresponding to several types of piano keys, or the first field-of-view image includes images corresponding to several types of piano keys and second-type piano keys. The smart musical instrument also includes a memory for storing the computer program; and a processor for executing the computer program and, when executing the computer program, implementing the piano key recognition method and / or the key recognition method based on key seams provided in any embodiment of this application.

[0138] For example, please refer to Figure 13 , Figure 13 This is a schematic block diagram of a computer device provided in an embodiment of this application. The above-described method can be implemented as a computer program, which can be used in, for example... Figure 13 It runs on the computer device shown. Figure 13 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory may include non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The computer programs include program instructions that, when executed, cause the processor to perform any key recognition method and / or a key-seam-based key recognition method. The processor provides computational and control capabilities to support the operation of the entire computer device. The internal memory provides the environment for the execution of the computer program in the non-volatile storage media; when executed by the processor, this program causes the processor to perform any key recognition method and / or a key-seam-based key recognition method. The network interface is used for network communication, such as sending assigned tasks.

[0139] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0140] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: S301, Obtain a first field-of-view image, which includes images corresponding to several types of piano keys; S302, Perform a first filtering operation on the pixels in the first field-of-view image based on the pixel values ​​to suppress first-type image noise, and obtain a second field-of-view image; S303, Perform a preset threshold processing on the second field-of-view image to obtain a third field-of-view image, which includes several target pixel sets, each consisting of several consecutive pixels with pixel values ​​lower than a preset threshold; S304, Perform a second filtering operation on the target pixel sets in the third field-of-view image based on a preset structuring element to suppress second-type image noise, and obtain a fifth field-of-view image; S305, Based on a second preset model, identify several of the aforementioned types of piano keys from the fifth field-of-view image.

[0141] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: S401, acquire a first field-of-view image, which includes images corresponding to several type I and type II piano keys; S402, based on the preset positional relationship between the type I and type II piano keys, and according to the piano key image region where the type I piano keys are located, extract a first partial image of the region where several type II piano keys are located from the first field-of-view image; S403, identify several initial key gaps between adjacent type II piano keys in the first partial image; and perform morphological operations on the several initial key gaps in the first partial image to obtain a target partial image; S404, identify several type II piano keys from the target partial image based on a third preset model; wherein, the morphological operation includes the following steps: S4031, acquire at least one target type II piano key adjacent to the initial key gap to be processed, and update the setting parameters of the structural element based on the perspective deformation parameters of the target type II piano key; S4032, perform morphological operations on the initial key gap to be processed based on the updated structural element.

[0142] For example, the processor is used to run a computer program stored in the memory, and is also used to implement the steps of the piano key recognition method and / or the key recognition method based on key seams provided in any embodiment of this application, which will not be repeated here.

[0143] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement the steps of the piano key recognition method and / or the key recognition method based on key seams provided in any of the embodiments of this application.

[0144] The computer-readable storage medium can be an internal storage unit of the smart musical instrument described in the foregoing embodiments, such as the hard drive or memory of the smart musical instrument. Alternatively, the computer-readable storage medium can be an external storage device of the smart musical instrument, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for filtering interference in piano key recognition, characterized in that, The method includes: S301, Obtain a first field-of-view image, wherein the first field-of-view image includes several images corresponding to a class of piano keys; S302, perform a first filtering operation on the pixels in the first field of view image based on the pixel values ​​in the first field of view image to suppress the image noise of the first form and obtain a second field of view image; S303, perform adaptive binarization processing on the second field image to obtain a third field image. The third field image includes a number of target pixel sets, which are composed of a number of consecutive pixels with pixel values ​​lower than a preset threshold. S304, perform a second filtering operation on the target pixel set in the third field of view image based on a preset structural element to suppress the second type of image noise and obtain a fifth field of view image; S305, based on the second preset model, identify several piano keys of the first type from the fifth field of view image.

2. The method according to claim 1, characterized in that, S303 includes: dynamically calculating an adaptive threshold based on regional illumination, performing preset threshold processing on the second field-of-view image based on the adaptive threshold, and obtaining a third field-of-view image that is a binary image.

3. The method according to claim 1, characterized in that, S303 includes: adding an illumination normalization layer before adaptive binarization and using contrast-limited adaptive histogram equalization processing.

4. The method according to claim 1, characterized in that S302 includes: Identify at least one pixel curve on the first field-of-view image, wherein the pixel curve is composed of multiple pixels and is used to reflect the degree of change in the pixel value of the pixel; Calculate the degree of pixel difference between at least one first target segment and an adjacent second target segment in the pixel curve, wherein the first target segment and the second target segment are composed of at least one pixel. Determine whether the degree of pixel difference exceeds the pixel difference threshold; If so, the first target segment is modified to reduce the degree of pixel difference; If not, retain the original pixel values ​​of the first target segment.

5. The method according to claim 1, characterized in that, S302 also includes: The pixel value distribution within the pixel neighborhood is analyzed, and the first field-view image is processed based on a multi-directional smoothing strategy, wherein the multi-directional smoothing strategy is one of median filtering, Gaussian filtering, or adaptive weight filtering.

6. The method according to claim 4 or 5, characterized in that, S302 also includes: Selective smoothing of the first field-of-view image includes replacing pixels whose difference from the neighborhood median exceeds a preset median threshold and which are not located at the edge of a piano key with neighborhood statistics, wherein the neighborhood statistics include either the median or the weighted mean.

7. The method according to claim 1, characterized in that, Prior to S303, it also included: adding chroma-weighted grayscale to the second field of view image, that is, enhancing the difference weight between the green and blue channels.

8. A smart musical instrument, characterized in that, The intelligent musical instrument includes: An image acquisition module is used to acquire a first field-of-view image, which includes several images corresponding to a class of piano keys. Memory, used to store computer programs; A processor is configured to execute the computer program and, in executing the computer program, implement the interference filtering method for piano key recognition as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The device includes: Memory, used to store computer programs; A processor is configured to execute the computer program and, in executing the computer program, implement the interference filtering method for piano key recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the interference filtering method for piano key recognition as described in any one of claims 1 to 7.

Images